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Description 

BACKGROUND OF THE INVEtfTlON 

1 . Fteld of the Invention; 

[0001] The present Invention relates to the field of 
graphical user Interfaces for computer systems and, 
more partlcularfy, to a method and apparatus for provid- 
ing an intuitive methodology for an operator to control 
remote devices with a computer controiied object-ori- 
ented user interface utiilzing animated graphic images. 

2. ArtBackgfOund: 

[0002] Most popular graphical user interfaces that are 
available today, such as the Xerox Starts, Apple Madn- 
toshO, and Microsoft Windows graphical user Interfac- 
es, are based upon the 'desktop metaphor.' In a desk- 
top metaphor system, the display screen Is treated as a 
virtual desktop. Graphk^ai symbols placed on the virtual 
desktop (i e. the display screen) are used to represent 
comnrK)n objects found in an office environment such as 
files, file fokJers, and printers. These desktop metaphor 
systems have several llmltattons which hamper the ef- 
fectiveness of the man-machine Interface between the 
human opemtor and the computer system. 
[0003] One problem with the existing desktop meta- 
phor graphical user interfaces is the use of a hierarchical 
fOe system. Most desktop metaphor graphical user in- 
terfaces are based upon a conr^uter having a file system 
in which the computer flies are organized in a hierarchi- 
cal structure. The hieFarchk:al file system structure has 
a root directory and several sub-directories within the 
root directory. Each sub-directory may have 8Uk)^b«di- 
rectories, and so on. The hierarchical file system struc- 
ture is graphically illustrated on the screen as some type 
of root file cabinet with file folders Inside. The Mk^rosoft 
Windows and Apple Madntoshid operating systems 
both use hierarchical file systems illustrated as file foM- 
ers. 

[0004] The hierarchical structure of file folders is often 
very confusing to new users. For example, many users 
get lost while navigating through hierarchk^l file sys- 
tems because the syst^ of keeping files in file f okierB, 
which are in otherfilefolderB nested to an arbitrary depth 
Is Inherently confusing. Users become lost because 
each file folder looks like every other file fokler, and the 
user Is provkied with little feedback as to his k>cation 
within the file system hierarchy. Also, each file folder 
.. .provides no infonfnatk>n as to what the file fotaier con- 
tains other than a user-defined lat>el. For examf)le, ex- 
cept for a short text label, whfeh may have been defined 
by another user, the user generally is not able to deter- 
mine whether an open folder is an accounting file folder 
containing spreadsheet data, or a data base file folder 
containing a customer data base. Therefore, navigating 
through the complex file system hierarchy is dHficuit 



since the cun^ent desktop metaphor graphical user In- 
terface offers little feedback as to a user's locatton within 
theffle hierarchy. 

[0005] In current graphk:al user interface systems 
5 based upon the desktop metaphor, certain applicatk>ns 
often do not fit well Into a desktop metaphor environment 
conceptually. Specifically, non-office related applica- 
tions may be Inappropriate for a system t>ased upon the 
desktop metaphor. For example, in a desktop metaphor, 
10 an appllcatk)n that controls a television wouki have to 
be placed into a "paper' file folder on the desktop. 
[0006] Another common problem with desktop meta- 
phor user Interfaces is that displaying a large number of 
appiteattons on the desktop results In a cluttered desk- 

is top. Many users of desktop metaphor systems keep 
large numbers of applcattons and files, usually In Icon 
fonn, on the display screen. Some attempt is made to 
use a graphfeal representation that klentifies the type of 
appllcatton or file. However, when a user wishes to open 

^ a partteular applteatton or file, the user must search the 
display screen to find the desired Icon Just as If the user 
were trying to find a particular paper file on a messy 
desktop, feons placed the desktop have no apparent hi- 
erarchy, and are all equal. This equality may confuse a 

25 user, and may create problems when the user wants to 
switch from one task to another. 
[0007] Several systems have evoh^ed to improve the 
organlzatk>n of the desktop metaphor graphfeal user in- 
terface. One system is the concept of "Rooms'. See, 

so Patrick Chan, 'Leambg Conskfetattons h User Intor- 
face Design: The Room ModeT, Report CS-84-1 6, Uni- 
versity of Waterk>o Computer Science Department, On- 
tario, Canada, July, 1984; Brenda laurel, "Computers 
as Theatre', Addlson-Wesiey Publishing Co., 1991; 

35 Brenda Laurel, "The Art of Hunnan-Computer Interface 
Design", Addison-Wesley Publishing Co.. 1990; U.S. 
Patent5,072, 412, Henderson, Jr. et al., 'Us&r Interface 
with Multiple Workspaces for Sharing Display System 
Otjeds't and the references cited therein. 

40 [0008] in systems based upon the Rooms concept, 
the user creates different rooms where the user per- 
fonms a single task or a group of related tasks. Each 
room represents an indivklual virtual desktop with a 
group of associated tasks. Each room is given a lat>el 

45 that Wentifies the room with the room's functions. For 
example, a user may create a mail room, a typing room 
with a word processor, and an accounting room with a 
spreadsheet Each room has only those icons and win- 
dows that are associated with the paftk:ular task of the 

50 room, as well as any windows shared by all rooms, such 
as a dock. 

[0OO9] To switch between tasks in a system that uses 
the Rooms concept, a user nuyves between the different 
rooms that are associated with the different tasks. For 
55 example, to go from a spreadsheet to retrieve email, a 
user moves from the accounting room to the mall room. 
To move between the two rooms, a user selects an icon 
that represents a door connecting the cunrent room to 
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the other room. Users can create doors between any of 
the available roonos. 

[0010] Although the Roonos concept helps solve the 
problem of the "cluttered desktop/ the Rooms concept 
still relies upon the common deslctop metaphor. For ex- 
ample, the Rooms interface concept does not help s\m- 
pllfy the confusing hierarchical file structure used to 
store information. The Rooms concept also does iitde to 
solve the fomi and function inconsistencies that are ap- 
parent when applications which do not fit into the desk- 
top metaphor are added to the system. 
[001 1] in prior art graphtoal user interfaces, each time 
a new piece of hardware is introduced to the system a 
new piece of control software must also be added by the 
user. For examplOp if a user of an Apple® Macintosh® 
or a user of Microsoft® Windows desires to add a new 
printer to the system, the user must also install a new 
printer driver to control the added printer. Therefore, 
each time a user desires to add a new device to the sys- 
tem, the user must also foltow a complteated procedure 
to add the appropriate software. 
[001 2] As will be described, the present invention pro- 
vides an improved graphical user interface that over- 
comes several of the problems associated with desktop 
metaphor systems. A method and apparatus for obtain- 
ing the user interface of a remote devk» without requir- 
ing the user to k>ad the related software is also dis- 
closed. The remote device can be controlled using the 
remote device's user interface as displayed on the 
present Invention's display screen. Although the user In- 
terface is deserved with reference to the control of re- 
mote devk:es, It will be appreciated from the discussion 
that foHows that the user interface may be applied to a 
variety of applk»tk>ns in computer woricstatlons, porta- 
ble computers, and hand hekJ control systems. 
[0013] Further examples of various prior art arrange- 
ments are described In EP-e-0439873 (IBM et aO and 
IBM Technteal Disclosure Bulletin, vol. 36. No. 4, April 
1d93. New Yoric. US, pages. 215-216. XP0a2012167, 
"Animated Command Execution loon". 

SUMMARY OF THE INVEMnON 

[0014] The present invention provkies an intuitive 
graphk:al user interface and a method for controlling re- 
niu>ted0vtees as set forth In the appended claims. 
[0015] The graphteal user Interface is displayed on a 
hand-held display devtoe. The hand-heki display device 
comprises at least one central processing unit (CPU) 
coupled to a graphk: display system. The graphfe dis- 
play system is touch sensitive to pemUtthe user to input 
infomnation using a fingertip wtthout the need for addl- 
ttonal Input devtoes such as a keyix>ard. A conrvnunica- 
tions circuit is also coupled to the CPU transmitting in- 
formation over some communication medium. The com- 
munications circuit is used to control and to communl* 
cate wfth renwte devices. 

[0016] The graphteal user interface creates a worid of 
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spaces organised Into a geographic map structure ttuit 
is simple to navigate through since each space provkies 
familiar geography sumMinds. Each space Is represent- 
ed by a background image whk:h provides navigational 
5 iandmarksthatidentifywherethespace is located within . 
the geographic map structure. For example, the back- 
ground image of a living room space contains items typ- 
ically found In a living room such as a lamp, a chair, and 
a table. Therefore, navigation through this "worid" is 

10 made easier because the user can orient himself by 
looking around and examining the sunroundings Just as 
a person would do in the real worid. 
[0017] Within each Space, the user interface provkies 
graphb objects that can be sele^ed and manipulated 

IS by the user. The graphfe objects are rendered as ani- 
mated cartoon-like graphk: objects that are easily kten- 
tifiable as real worid objects such as televisk>ns and 
VCRs. There are at least three types of objects that exist 
in the Spaces of the user Interface: data objects, but- 

^ tons, and portals. Data objects are objects that are as- 
sociated with a particular piece of data. Buttoris are ob- 
jects that are associated with a particuiarfunction or ac- 
tion; when a user selects a button, the user interface 
perfonns ttie function or action associated witti the but- 

^ ton. Portals are objects that are displayed ki a first 
Space but are assodated witii a second Space; when a 
user selects a portal the user passes through the portal 
and moves to the second Space associated with that 
portal. The user environment Is further enhanced by the 

30 use of sound, coordinated wtth every user action and 
supplementing the visual feedback provided. 
[001 8] To control real worM renriote devices, many of 
the graphto objects presented on the screen are asso- 
ciated witi) remote devfees In the real worid. To control 

S5 a remote device, a user selects the graphic object pic- 
tured on the display screen associated witti the particu- 
lar remote devtee. Afterttie user selects a graphic object 
associated witii a remote devtee, the hand-heM display 
device displays a user interface for controlling the re- 

40 mote devbe. The user can then interact witii the remote 
device's user interface. To control the remote device, the 
present invention Invokes a device driver method forthe 
remote device. The devk^ driver metiiod may be locat- 
ed within the remote devfee or within the local hand-heki 

45 display device, depending on the remote device's so- 
phistication. 

BRIEF DESCRIPTION OF THE DRAWINGS 

so [0019] The objects, features and advantages of the 
present Invention will be apparent to one skilled in the 
art, in view of ttie following detailed description In which: 

noURE la ilhistrates an overview of the remote 
55 control functionality provkled by the preseni Inven- 
tion. 

RQURE 1 b Illustrates an overview of tiie hardware 
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used to Implement the hand-held display device of 
the present invention. 

RGURE 1c illustrates a block diagram of an Intelli- 
gent remote device that is controlled by the present 5 
Invention. 

nGURE Id lltustrates a block diagram of a simple 
remoie device that is controlled by the present in- 
vention. 10 

RGURE 1e Illustrates a block diagram of conven- 
tk>nal electronic device that is controOed remotely 
by the present Invention. 

1$ 

RGURE 2a illustrates a structure of different geo- 
graphic spaces. 

RQURES 2b - 2e Illustrate a sample geographic 
map structure of spaces used to provkie a user in- ^ 
terface. 

RGURE 3 illustrates a view of the outer-space 
Space of the user interface based upon the geo- 
graphk: map structure in Figures 2b and 2c. ^ 

RGURE 4 illustmtes a first view of the Earth's sur- 
face Space in the user interface based upon the ge- 
ographic map structure in Rgures 2b and 2c. 

30 

RGURE 5 Illustrates a second view of the Earth's 
surface Space In the user interface based upon the 
geographic map structure in Figures 2b and 2c. 

FIGU RE 6 Illustrates a view of the first house Space S5 
of the user interface based upon the geographk: 
map struc^re In Rgures 2b and 2e. 

FIGURE 7 Illustrates a view of a living room Space 
in the user interface based upon the geographic 40 
map structure in Rgures 2b and 2c. 

RGURE 8 Illustrates a view of a televiskm Space in 
the user interface based upon the geographto map 
stnjcture in Figures 2b and 2e. ^ 

RGURE 9 IDustrates a view of a television program 
listing book Space in the user interface based upon 
the geography map structure In Figures 2b and 2c. 

50 

RGURE 10 Illustrates a view of a single television 
program description Space of the user interface 
based upon the geographic map structure in Rg- 
ures 2b and 2c. 

55 

RGURE 11 illustrates a conceptual view of the tel- 
evlston program listing book Space of Rgure 9. 



RGURE 12 illustrates a view of a plurality of objects 
which do not nt on a single screen display. 

RGURE 1 3 Illustrates a conceptual view of the plu- 
rality of objects placed on an object wheel. 

RGURE 14 Olustrates a conceptual view of the ob- 
ject wheel of Rgure 13 where the object wheel has 
rotated revealing new ofcjects. 

RGURE 1 5 Illustrates the Spin gesture being used 
to spin an object wheel. 

FIGURE 16a illustrates a Ptokup gesture being 
used to pickup an object off an object wheel. 

RGURE 16b illustrates a Pk:kup gesture being 
used to pickup an object from a Space. 

RGURE 17a Illustrates the Agent Portal of the 
present lnventk)n wearing a doctor's uriiform to 
symbolize a diagnostic mode. 

RGURE 17b illustrates the Agent Portal of the 
present inventfon hokiing a pencil to symbolize a 
wrttingmode. 

RGURE 17c Hlustrates the Agent Portal of the 
present Inventfon altervipting to alert the user about 
a recent event 

RGURE 18a illustrates an intelligent device trans- 
mitting its graphical user interface to a display de- 
vice. 

RGURE 18b mustnates a display devtee displaying 
the graphical user interface of an bitelligent control- 
lable devtoe. 

RGURE 16c illustrates the display device transmit- 
ting user interactions with the user interface to the 
intelligent device. 

RGURE 18d lllustrBtes the intelligent devk:e trans- 
mitting return values or feedback to the display de- 
vice. 

RGURE 1 9a illustrates a conceptual diagram of the 
program ot>jects in the intelligent device being sent 
to the display device. 

RGURE 19b illustrates aconceptual diagram of the 
program objects of the intelUgent device In the dis- 
play device. 

RGURE 19c illustrates aconceptual diagram of the 
display devk» Invoking a method in an ol^ect within 
the remote inteUigent device. 
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nGURE 19d illustrates a conceptual diagram of the 
method In an object within the remote intelligent de- 
vice returning status. 

FIGURE 20 illustrates a flowchart of the transfer of 
an Intelligent device's user interface. 

FIGURE 21a lllustn^tes a state diagram of the RFC 
manager for a client requesting a remote method 
invocation. 

RGURE 21 b illustrates a state diagram of the RFC 
manager for a seiver handling a remote method in- 
vocation. 

FIGURE 21c Illustrates the sequence of a remote 
method invocation. 

FIGURE 22a illustrates the simple device transfer- 
ring its graphical user Interface to a display device. 

RGURE 22b Illustrates a display device displaying 
the graphicai user interface of the simple controlla- 
ble device. 

FIGURE 22c illustrates the display device transmit- 
ting simple control codes to the simple device. 

RGURE 22d illustrates the siniple device transmit- 
ting return values or feedbacic to the display device. 

RGURE 23a Illustrates a conceptual diagram of the 
program objects In a sbfnple device being sent to the 
display device. 

RGURE 23b Illustrates a conceptual diagram of the 
program objects of the sinpie device in the display 
device. 

FIGURE 23e iOustrates a conceptual diagram of the 
display device involdng a method in a program ob- 
ject and transmitting control codes to the simple de- 
vice. 

RGURE 23d illustrates a conceptual diagram of the 
remote simple device retuming status infonnatlon. 



convey the substance of their woric to others sldlled In 
the art. 

[D021] An algorfthm Is here, and generally, conceived 
to be a consistent sequence of steps leading to a desired 

5 result. These steps are those requiring physical man^ . 
ulations of physical quantities. Usually, though not nec- 
essarily, these quantities taice the fonn of electrical or 
magnetic signals capable of being stored, transferred, 
combined, compared, selected, chosen, modified, and 

10 otheiwlse man^ulated. It proves convenient at tb^es, 
principally for reasons of common usage, to refer to 
these signals as bits, values, elements, symbols, char- 
acters, images, tenms, numbers, or the ilice. It should be 
borne in mind, however, that all of these, as wen as sinv 

15 liar tenns, are to be associated with the appropriate 
physical quantities and are merely convenient labels ap- 
plied to these quantities. 

P022] In the present case, the operations are ma- 
chine operations perfonned in conjun^on with a human 

^ operator. Useful machines for perf onning the operations 
of the present Invention include general purpose digital 
computers or other similar devices. In all cases, the dis- 
tinction k>etween the method operations of operating a 
computer and the method of computation itself should 

25 be Icept in mind. The present invention relates to nnethod 
steps for operating a computer and processing electrical 
or other physical signals to generate other desired phys- 
ical signals. 

IP023] The present invention also relates to appara- 

90 tus f or perf onning these operations. This apparatus may 
t>e specially constructed for the required purposes or it 
may comprise a general purpose computer selectively 
activated or reconfigured by a computer program stored 
in the computer. The algortthms presented herein are 

S5 not inherently related to any particularcornputer or other 
apparatus. In particular, various general purpose ma- 
chines nr>ay be used with programs in accordance with 
the teachings herein, or It may prove more convenient 
to construct more specialized apparatus to perform the 

40 required method steps. The required structure for a va- 
riety of these machines will appear from the description 
given below. Machines which may perf omn the functions 
of the present Invention Include those manufactured by 
the Assignee, RrstPerson, inc., as well as other manu- 

45 facturers of computer systems, including Sun Microsys- 
tems, inc. of Mountain View, California. 



RGURE 24 Illustrates a flowchart of the transfer of 
a simple device's user Interface. 

. NOTATION AND NOMENCLATURE 

[0020] The detailed descriptions which follow are pre- 
sented largely in terms of display images, algorithms, 
and synf)t)olic representations of operations of data bits 
within a computer memory. These algorithmic descrip- 
tions and representations are the means used by those 
skilled in the data processing aits to most effectively 



COPING DETAILS 

[0024] No particular programming language has been 
Indicated for carrying out the various procedures de- 
scribed herein. This is due in part to the fact that not ail 
languages that might be mentioned are univereaily 
available. Each user of a particular computer will be 
aware of a language which is most suitable for his inrv 
mediate purposes, in practice, it has proven useful to 
siA)stantially Implement the present invention in a high 
level language which is then complied Into machine ex- 



50 



55 



5 



9 



EP0626635B1 



10 



editable object code. Because the computers and the 
monitor systems which nnay be used In practicing the 
Instant Invention consist of many diverse elements, no 
detailed program listing has been provided, it is consid- 
ered that the operations and other procedures de- 
scribed herein and illustrated In the aocon^Muiylng 
drawings are sufficiently disclosed to penult one of or- 
dinary sidll to practice the instant invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0025] A computer based remote control system with 
an intuitive graphical user interface is disclosed. In the 
following description, for purposes of explanation, spe- 
cific nomenclature is set forth to provide a thorough un- 
derstanding of the present Invention. However, it will be 
apparent to one skilled in the art that these specific de- 
tails are not required to practice the present invention. 
In other instances, well known circuits, functk>ns, proc- 
esses, and devices are shown In block diagram and con- 
ceptual diagram fonr) in order not to obscure the present 
InventkNi unnecessarily. 

System Overview 

[0026] Figure la illustrates an overview of the com- 
puter based remote control system of the present Inven- 
tk>n. The main component of the present invention con- 
sists of a hand-held display devk» 170 as illustrated In 
Rgure la. In the present embodiment, the hand-held 
display device 170 comprises a conrputer system with 
a cok)r graphic touch screen display. The hand-held dis- 
play device 170 is designed to control any compatible 
renwte devk» such as thenmostat 150, vkieo cassette 
recorder 39, and stereo system 152. The hand-held dis- 
play devfee 170 communicates with the remote devices 
using a conrmiunications medium 160. 
[0027] The remote control system of the present In- 
vention as illustrated In Rgure la differs slgniftoantly 
from prior art remote control systems since it is based 
upon modular object-oriented software. The object-ori- 
ented software provkies features prevtously not availa- 
ble in prior art remote control syst^ns. For example, 
when an operator (sometime referred to herein as a "us- 
er^ adds a new compatible remote device, such as ther- 
mostat 150 to the system, the new remote devbe trans- 
mits infonmation about itself to the hand-held display de- 
vtee 170. The transmitted information Includes a user 
interface object that can be used to render a graphical 
user Interface for the remote device. The present inven- 
tion uses the xiser interface object to display a graphical 
user Interface for the new remote devtee on the hand- 
held display devk» 170. The transmitted Infomiation 
may also include a device driver object that is used to 
control the new remote device. To organize all the avail- 
able user Interface objects, the hand-heki display device 
170 creates a "World" of geographic Spaces that can be 
easily be navigated using a set of gestures. 



[0028] This Specification will first disclose the hard- 
ware required to construct the hand-heki display device 
170 and the compatble renrM>te devices of the present 
invention. The graphk^l user interface corr^rlsing geo- 

5 graphic Spaces and objects which is displayed on the : 
hand-hekJ display device 170 wiU then be described. A 
set of gestures used by operators to navigate around 
the user interface of the present Invention wiH also be 
defined. To combine the user interface elements, a tour 

10 through an embodiment of the user Interface of the 
present invention will be provided to iiiustrate how the 
navigational gestures are used to Interact wtth the ob- 
jects displayed on the hand-heki display device 170. Fi- 
nally, the system of obtaining infonnatton from the re- 

15 mote devices Is defined, such that the hand-held display 
device 170 may communicate with and control remote 
external devices. 

[0029] Due to the RmllBttons of a written Specifteatkm 
to describe animated events on an Interface, the reader 

20 Is refenred to a vkleo tape entitled "Green Project User 
Interface," submitted by the applicant concunrentJy with 
the filing of the appUcatton on which this patent is based, 
in addition it must also be noted that the features of the 
present invention are illustrated ki black and white within 

25 the accompanying figures. However, in the presently 
preferred embodiment, and as described bekyw, objects 
and features are displayed ki cok>r. 

Hardware Cenflfluratlon 

30 

[0030] The hand-heM display devk:e 170 of the 
present inventton comprises a conputer system, such 
as that illustrated in Figure lb. The computer system 
170 of Figure lb Includes a central processing unit 

35 (CPU) 30, a memory unit 33, and an input/output (I/O) 
unit 34. These elements are those typically found in 
most general purpose computers, and in fact, the hand- 
held computer system 170 is kitended to be represent- 
ative of a broad category of data processing devtees ca- 

40 pable of generating graphic displays. 

[0031] Coupled to the Input/output (I/O) unit 34 of 
computer system 170 is a color graphic screen display 
37. The screen display 37 comprises, in part, a color 
raster display devk:e such as a cathode ray tube (CRT) 

45 nrK)niU>ror a OqukI crystal display (LCD). The screen dis- 
play 37 must be oX suffteient resolution such that screen 
display 37 can render graphte images. 
[0032] In the prefenred embodiment, the screen dis- 
play 37 further comprises a touch screen disptey sys- 

so tern. The touch screen display Inchides a feature that 
can detect the presence of a finger touching the display 
screen 37. Furthennore, the touch screen can Mentny 
the precise kx^ation of the finger touching the display 
screen 37 such that the display screen can sense finger 

55 gestures made by a user's finger on the display screen 
37. The touch screen display 37 may comprise one of 
a variety of touch sensitive display screens commercial- 
ly available on the nuuicet 
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[0033] Alternatively, the hand-held computer system 
170 may be constructed with a conventional graphic dis* 
play system without a touch sensitive display screen. 
Such a system requires an alternative input system to 
obtain the user's interactions. Figure lb illustrates an 
alternative Input device 42. Although the user Interface 
of the present invention may be used with alternative 
input devices 42 such as mice, tracldballs, and Joystidcs, 
the system will be described within this specification as 
the preferred emt>odiment with a touch screen display 
37. 

[0034] in operation, the CPU 30 generates and dis- 
plays a graphical user interface on touch screen display 
37 through the execution of computer code stored in 
memory 33. The computer code stored In memory 33 is 
not disclosed herein since the description of the present 
Invention in this Specification is sufficient for one sidlled 
in the computer art to utilize the teachings of the Inven- 
tion in a variety of computer systems using one of many 
computer languages. 

[0035] Coupled to the input/output (t^) unit 34 of 
computer system 170, In Figure lb, is an external com- 
munication circuit 35. The external communication cir- 
cuit 35 provides a system for transmitting and receiving 
infomnatlon to and from the hand-held computer system 
170. The present invention uses the external communi- 
cation circuit 35 in cornputer system 170 to communi- 
cate with remote devices such as a video cassette re- 
corder 39 and a thenmostat 150, across communica- 
tions medium 160. The communication circuit 35 may 
use one of many well-icnown communication technolo- 
gies such as radio waves, low power, cellular, infrared 
signals, or data signals canied on existing power lines. 
[0036] To be controlled by the present hand-held dis- 
play device 170 of the present invention, a renrwte de- 
vice must have at least one method for receiving infor- 
mation transmitted by the hand-held display deyice 170. 
ideally, the renrK>te devices are intelligent devices thai 
can both transmit and receive information. However, the 
hand-held display device 170 can control any remote 
device that fits within any of the three following renrtote 
device "types". 

[0037] The first type of remote device that can be con- 
trolled by the present invention is an "intelligent remote 
device" spedfically designed to operate with the hand- 
held display device 170. in the present invention, an "In- 
telligent remote device" Is defined as a remote device 
that contains a microprocessor, a communication sys- 
tem, and a men)ory. Rgure 1c provides a biocic diagram 
of the connponents in an IntelBgent remote device 150. 
~ As illustrated m Figure 1c, the intelligent remote device 
150 comprises a CPU 403, a memory 404, a communi- 
cations drcuit 401, and a device control circuitry 405. 
[0038] The hand-held display device 170 of the 
present invention conwnunlcates with an inteillgent re- 
mote device 150 through the intelligent remote device's 
conununications circuit 401. The memory 404 stores 
programs and data used by the CPU 403, including a 
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user Interface object 303 and a device driver object 301 . 
The CPU 403 in the intelligent remote device invoices 
methods in the device driver object 301 to control the 
remote device. 

5 [0039] The second type of remote device that can be . 
controlled by the present invention is a 'simple remote 
device" spedfically designed to operate with the hand- 
held display device 170. In the present invention, a re- 
mote device is defined as a "sfarnpie remote device" if 

10 the remote device does not have a microprocessor or 
the remote device contains such a primitive processor 
that the hand-held display device 170 can not invoice 
methods In a device driver object within the remote de- 
vice. 

15 [0040] Hgure Id provides a block diagram of the 
components in a typical simple renx>te cievice 155. As 
illustrated in Rgure Id, the simple remote device 155 
comprises an emitter 420, a receiver 421, a memory 
424. and device control circuitry 425. The emitter 420 in 

20 the simple remote device 155 Is used to transmit the 
contents of the memory 424 to the hand-held display 
device 170. Therefore, the simple renwte device 155 
transmits both the user interface object 353 and the de- 
vice driver object 351 in the memory 424, such that the 

25 hand-held display device receives both objects. 

[0041] To control the simple remote device 155. the 
display device 170 invoices a method within the device 
driver object 351 to translate the user's interactions into 
simple predefined control codes. The display device 170 

30 then transmits the simple predefined control codes to 
the simple remote device 1 55 which receives the control 
codes though receiver 421 and passes the infonnation 
to the device control drcuitry 425. 
[0042] Finally, the third type of remote device that can 

S5 be controlled by the present Invention is a conventional 
electronic device that can be controlled by transmitted 
signals such as infrared signals. Therefore, the present 
invention is baclcward connpatibie with most existing de- 
vices that can be controlled remotely. Figure 1e ilius- 

40 trates a typical electronic device 157 that can be con- 
trolled by coded signals. The device has a receiver 450 
for receiving transmitted coded control signals and de- 
vice control drcuitiy 455 for responding to the coded 
control signals. The hand-held display device 170 con- 

45 trols the electronic device 157, transmitting coded 
control signals. The remote electronic device 157 re- 
ceives the transmitted signals and responds according- 
ly. 

50 The Display Device User Interface 

[0O43] In order to create a new product that Is easy to 
use, product designers attempt to malce the new product 
operate in a manner similar to an old product that the 
55 Intended operator already icnows how to operate. For 
example, audio cassette tape recorders have been 
available for such a long time that most people Icnow 
how to operate an audio cassette tape recorder. The 
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play, stop, rewind, fest>forward, and record buttons on 
cassette tape recorders are therefore familiar to niost 
people. Video cassette recorder manufacturers pro- 
duced easy-to-use video cassette recorders by taidng 
advantage of this common Icnowiedge. The video cas* 5 
sette recorders are easy to use since the video cassette 
recorders used the same buttons. I.e., the play, stop, re- 
wind, fast-forward, and record buttons, that audio cas- 
sette recorders utilize for operation. 
[0044] To malce the hand-held display device 170 io 
easy to use, the present invention displays an intuitive 
graphical user interface on the hand-held display device 
170. This section defines the graphical user interface 
which is provided to users on the hand-held (fisplay de- 
vice 170. 15 
[0045] To design a graphical user interface that is 
easy to navigate, designers should organize the graph- 
ical user interface with a structure that most users al- 
ready understand. Humans live in a three dimensional 
geographic world. To survive In our three dimensional ^ 
geographic world people must learn how to navigate 
around the local geographic area, including such geo- 
graphic spaces their house, their neighborhood, and 
their town. In order to do this, each person creates a 
mental map of geographic spaces of ttiebr relevant local 25 
world. 

[0046] The mental maps of geographic spaces people 
create are de-facto geographic map structures. Figure 
2a illustrates a typical geographic map structure. For ex- 
ample, every person on this planet knows that they live 30 
on the Earth. Each person also lives in a certain town; 
yet, each person is aware that other towns exist on the 
Earth. Each person lives at a particular house on a par- 
ticular street; yet, each person Is aware that there are 
other streets in the town with other houses on each 35 
street. The identification of one particular house on one 
street within one town on Earth is a naturally occurring 
geographic n\ap structure as illustrated In Figure 2a. 
[0047] The geographfc map structure can be extend- 
ed into the house. For example, the Hving room is one 40 
room of many within ttie house. The television, televi- 
sion program schedule and the video cassette recorder 
are objects wlttiin tt)e living room. Each object can have 
its own geographic space. The map structure can also 
be extended to at>stract spaces as wen as geographic ^ 
spaces. Each television program listed in a television 
program schedule has a program description. The pro- 
gram description can be considered a space within the 
television program schedule. Most people understand 
both the structure of geographb spaces and the detail 50 
that will be found wittiln each geographic space. There- 
fore the de-facto geographic map structure of spaces 
represents a structure that is familiar to most people. 
[0048] The present invention uses the familiar geo- 
graphic map structure of our three dimensional world In 55 
a graphical user interface. The present Invention de- 
scribes a technique of user navigation bi an arbitrarily 
large and complex geographic map structure through 



graphic bacl^reund Images that depict geographic 
spaces. Such geographic map structures improve a us- 
er's memory of complex data organization. The advan- 
tage of including such geographic map structures is that 
the familiar structures enable a wider range of people to 
use infonnation storage facilitated by a computer. 
[0049] Figures 2b and 2e Illustrate a sample geo- 
graphic map structure utilized by the present Invention. 
In the present invention, each Space in the geographic 
map stmctures of Rgures 2b and 2c is represented as 
a bacicgreund image on the display screen of hand-held 
display device such as ttie one Olustreted in Figure lb. 
Located wittiln each geographic Space are objects ttiat 
can be selected and manipulated by the user. 

User interface Display Elements 

[0050] The user interface environment of the present 
Invention consists of two main display elements: "Spac- 
es" and selectable "objects." Each area of tiie geograph- 
ic map structure of Figures 2b and 2c represents a 
Space. The Space provides a geographic reference that 
informs a user of the user^ cun^nt location wtthln the 
geographic mep structure. Wlttiin each Space are one 
or more "objects." A user may select and manipulate the 
objects within each Space. In the present embodiment 
of the hand-held display device 170 with touch screen 
display 37, a user may select objects by touching the 
objects displayed on the touch screen display 37. Point- 
ing devices olher than the user's finger and biput devic- 
es, other than a touch screen, might handle selection 
and user input in a different manner. 

Spaces 

[0051] Each Space in Figures 2b and 2c has an as- 
sociated bacl(ground image that contains appropriate 
Imagery for the Space. When a user entere a Space, the 
Space's background image is rendered on the display 
screen 37. Each background im^ contains landmarks 
that ho\p the user Uentify where the user is. For exanv 
pie, the Earth's surface Space 32 of Figure 2b Is lllus- 
treted as the background Images In Figure 4 and Figure 
5. The Earth's surface background Image of Figures 4 
and 5 resembles a street on ttie Earth's surface, thereby 
informing a user that the user Is at the Earth's surface 
Space of thei map structure. The system renders the 
background image using desaturated or dimmed colors 
such that the background Image can easily be distin- 
guished from Selectable objects. The t)ackground im- 
age for a Space may be larger than the display screen 
37; therefore, the present Invention provkies a gesture 
for "panning" such that a user may move around wttiiin 
a Space as will be explained later. 

Objects 

[0052] Objects are indlvklual graphteal loons ttiat are 



8 



15 



EP0626G35B1 



16 



rendered within the bad^round images of Spaces. 
Each object is actually an object-oriented programming 
type of object containing data methods which can be in- 
voiced to act on the data. A user can select and manip- 
ulate any of the objects which appear within the back- 
ground image of a Space. Each object is brightly colored 
and outlined in black. The bright colors distinguish the 
objects In a Space from the Space's background image 
that Is drawn using desaturated colors. When a user 
points to an object, the user interface generates a 'Se- 
lectable Object" sound that Informs the user that the ob- 
ject can be selected and manipulated. Referring again 
to Rgure 4, in the present example, the houses 47 and 
48 are objects that can be selected and manipulated by 
a user. 

[0053] An 'anticipation animation' is used In the 
present Invention to inform the user when some action 
Is taking place. The anticipation animation is a short an- 
imation displayed on the display screen 37 that is gen- 
erated by the CPU 30. In most cases, the anttelpation 
animation consists of enlarging an object's feon such 
that the Icon appears to nnove closer to the user when 
the object is selected. 

[0054] For example, refenlng to Figure 3, when the 
Earth portal 100 is selected, the Earth portal 100 ex- 
pands such that It appears to nrK)ve doser to the user. 
After the Earth portal 1 00 anticipates, the user Interfaces 
move the user to the Earth's surface Space 32 that Is 
associated with the Earth portal 1 00 as illustrated In R9- 
ure 4. 

[0055] When a user selects an object by pointing to 
the object for a predetemnined minimum period of time, 
a sound Is generated, the object antk^ates, and the 
CPU invokes a method within the object whfeh performs 
an actbn associated with the object. 
[0056] There are at least three types of objects that 
exist in the user interface of the present embodimer^ 
data objects, buttons, and portals. The simplest object 
Is a data object. Data objects are objeds associated with 
a set of data. Buttons are objects that provide a means 
for accomplishing functions. Portals provkie a method 
for moving between different Spaces. Each type of ob- 
ject win be explained indhridually bekyw. 

Data Objecte 

[0057] A data object is a simple object used to repre- 
sent some piece of information. For example, a data ob- 
ject may represent a color or a television channel. The 
data objects have no methods that can be Invoked to 
perfonn an associated action or function when the data 
object is selected. However, a data object can be 
dropped onto other objects that use the Inf onnatlon in 
the data object. For example, a television program list- 
ing data object can be dropped onto a video cassette 
recorder object to program the video cassette recorder 
to record the televisk>n program. 



Buttons 

[0058] A button object Is a spedai object that has an 
associated action or function. When a user selects a but- 

5 ton, the CPU 30 Invokes the action method in the button 
object to cause the executton of the action or function 
assodated the button. When the button is selected, the 
CPU also antkHpates the button and generates a 'button 
adion' sound provkling feedback to the user that the 

10 button has been seleded. Certain buttons provide ad- 
ditional visual feedback to Infomi the user of the results 
of the method Invoked by the CPU to perfomn the actkMi 
assodated with the button. 

[0059] For example, refenlng to Figure 8, a teieviston 

IS power 'on' button 61 and a television power "ofT button 
62 are displayed. When a user selects the televlsk)n 
power "on" button 61, the television power 'on' button 
61 anticipates and the CPU 30 generates the 'button 
action" sound. The CPU 30 also Invokes a method in 

so the button objed that transmits the necessary Infonna- 
tfon to turn on a real televisk>n set assodated with the 
television control panel Illustrated In Figure 8. 
|P060] If the real televisbn s^ is an 'intelligent remote 
device", the television set may retum a result value to 

^ the display device informing the user whether or not the 
television was turned on. The user Interface may then 
light up the television power "on" button 61 and generate 
a 'hum' sound to Infomn the user that the real television 
was actually tumed on. 

30 pool] Buttons can be copied and moved to different 
Spaces so that a user can customize the "workT of the 
user interface. For example, the television power "on" 
61 and the television power "ofT 62 buttons could be 
copied, and the button copies nrtoved from the televlsk>n 

35 contrdSpacetoany Other Space. A user may use the 
copies of the television power "on" 61 and the teievlskm 
power "OfT 62 buttons in the new Space to control the 
televisk>n. 

40 Portals 

P062] Portals are objects that provkie a simple meth- 
od of moving from the cunrent Space to another Space 
associated with the portal. A user moves from the cur- 

45 rent Space to another Space assodated with a portal by 
simply Seleding the portal. After a user selects a portal, 
the portal antteipates, generates a "movement through 
portal sound," and the CPU Invokes a method that 
moves the user to the Space assodated with the portal. 

so [0063] For example, refenlng to the geographk) map 
of Rgure 2b, to move from the Earth's surface Space 
32 to the house Space 50, a user simply selects the 
house portal 48. Referring to the Earth's surface Space 
32 as Illustrated in Figure 4, a user performs the move 

ss by selecting the house portal 48 within the Earth's sur- 
face Space 32. After being seleded, the house portal 
48 antidpates, generates a "movement through portal 
sound," and then the user is moved to the house Space 
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50 as illustrated In Hgure 6. 
[0064] Portals can be duplicated and moved such that 
a user can customize the "world" of the user interface. 
For example, In Figure 7 a television program schedule 
portal 71 Is illustrated within the Ih^lng room Space 60. 5 
The TV program schedule portal 71 Is associated with 
a Space that contains TV program schedule infomia- 
tion. A user can make a copy of the television program 
schedule portal 71 and move the copy to another Space 
where the user would like to be able to access the tele- io 
viskm program schedule. The user can then use the 
copy of the television program schedule portal to access 
the same information that could be accessed using the 
original television program schedule portal. Thus, a user 
would be able to access the same infonmatton from dif- is 
ferent Spaces. 

[0065] There are two special portals that are almost 
always available to cm operator of the user interface: a 
Wayback portal and an Agent portal. 

20 

The Wayback Portal 

[0066] The Wayback portal provkies a way for a user 
to trace back a path that the user has traveled along 
between different Spaces. The Wayback portal 45 con- 2s 
sists of a small rectangular frame that encbses a snap- 
shot of the previous Space along the path of Spaces 
that the user traveled along. Since the background im- 
age for a Space can be larger than what can be seen in 
a single view, the Wayback portal shows a snapshot of so 
the view of the previous Space exactly as it appeared 
when the user went through a portal in that previous 
Space. In the present embodiment, the Wayback portal 
is available to the user In every Space In the lower rfght- 
hand comer of the display screen. If desired, the user S5 
can move the Wayt)ack portal to a new locatton on the 
display screen such that the Wayback portal wiD then 
appear at the new locatton on the screen in every Space. 
[0067] To move one step backward along the cun^nt 
path,theuserSelect8theWaybackpoital45.WhenSe- 40 
lected, the CPU 30 anticipates the Wayback portal 45, 
generates a "move through Wayt>ack portal" sound, and 
moves the user to the Space pictured within the Way- 
back portal. In the present embodiment, the "move 
through W^iyback portaT sound is actually the same 43 
sound as the "n^ove through portar sound except that 
It is played backwards. By playing the "move through 
portaP sound backward when moving through the Way- 
back portal, the movement backward ak>ng the current 
path is emphasized to the user. ^ 
[0068] As users go through portals they leave a trail 
of invisible markerB. Each marker represents a visit to 
a Space. The Wayfc>ack manages these markers and 
maintains them in the order that the Spaces are visited. 
The markers and the markers* order define the path that ^ 
the user has taken through the work! of the user Inter- 
face. Thus, when a user goes though the W^iyback, the 
path defined by the markers is retraced one marker at 



a time. Each time the user goes through the Wayback 
to a Space, a marker is removed from those cun^ently 
managed by the Wayt>ack. 

[0069] For example, referring to geographic map 
stru^ure Illustrated in Figure 2b, if a user starts in the 
outer-space Space 09, moves to the Earth's surface 
Space 32, and then moves to the house Space 50; a 
path will be generated from the outer-space Space 09 
to the Earth's surface Space 32 to the house Space 50. 
After traversing this path, the display screen 37 will dis- 
play the house Space 50 as depicted in Figure 6. In the 
kywer right comer of the house Space 50 Is the Wayback 
portal 45. As Illustrated in Rgure 6, the Wayback portal 
45 displays a miniature snapshot of the prevtous Space 
along the Space path. In this case, the prevtous Space 
ak>ng the Space path is the Earth's surface Space 32; 
therefore, the Wayback portal 45 displays a miniature 
snapshot of the Earth's surface Space 32 as It appeared 
when the user went through the house portal. If the user 
then Selects the Wayback portal 45. the Wayt>ack portal 
45 antk:ipates, generates the "move through Wayback 
portal" sound, and then nnoves the user to the Earth's 
surface Space 32 as depteted In Figure 4. At the Earth'b 
surface Space 32 as deleted In Hgure 4 the Wayback 
now displays a miniature snapshot of the outer-space 
Space 99, whk^h was the first Space ak>ng the path. The 
Wayback portal 45 therefore provides a "way back" 
ak)ng the path of Spaces the user has traveled. 
[0070] The present Invention Implements the Way- 
back portal 45 as a stack of data structures, wherein 
each data stnjcture contains a pointer and a pair of co- 
ordinates. The pointer In each data structure on the 
stack points to a Space that the user has prevk>usly vis- 
ited. The pair of coordinates in each data structure on 
the stack specifies exactly where the user was within 
the Space pointed to by the pointer when the user left 
the Space through a portal. Each time the user nnoves 
through a portal from a particular Space, the user inter- 
face "pushes" a data structure, containing the pointer to 
the Space and the coorcfinate pair whfch defines exactly 
where the user was within the Space when he went 
through the portal, onto the wayback stack. The Way- 
back portal 45 displays a miniature snapshot of the 
Space that is pointed to by the pointer In the data struc- 
ture on the top of the stack. To trace back along the path 
of Spaces, each time the user Selects the Wayback por- 
tal 45, the user interface "pops' a data structure off the 
stack and moves to the Space indicated by the pointer 
In the data structure. 

The Agent l^ortal 

[0071] The Agent Is an anthropomorphk: character 
that aids the user in navigating around the user interface 
and in dealing with events that occur in the "workf cre- 
ated by the user interface. Refening to the Earth's sur- 
face Space of Figure 4, the Agent 41 is Olustrated in the 
lower left comer. ITie Agent 41 carries objects around 
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for the user. This carrying function Is represented by a 
briefcase, which the Agent also carries. A user can ac- 
cess objects that the Agent Is canying by Selecting the 
Agent portal 41 thereby entering an Agent Space asso- 
ciated with the Agent portal 41 . When a user Selects the 
Agent portal 41, the Agent anticipates by appearing to 
pull out the briefcase and open the briefcase up. The 
CPU then generates the "move through portal" sound 
and moves the user to the Agent Space. 
[0072] To place an object into the Agent* s briefcase, 
a user picks up the object and then "Drops" the object 
onto the Agent portal 41 . The object Dropped onto the 
Agent portal 41 is then always accessible to the user 
from any Space by entering the AgenTs Space. The 
Agent Is discussed more fully below in the section enti- 
tled The Agent." 

Object Wheel 

[0073] It is often necessary that a user malce choices 
among an large, arbitrary number of objects. Therefore, 
the user interface must provide both a method of dis- 
playing a large number of objects and a method of al- 
lowing the user to Select any of the displayed objects. 
[0074] For example, referring to the television control 
Space 64 illustrated In Rgure 8, the user interface of 
the present invention must provide a method of display- 
ing an arbitrary number of television channels for the 
user to choose from. In any system, screen space is at 
apremlum, so when the user must choose among many 
possible options, the options must be presented conv 
pactly. In the present invention, if there are 500 televi- 
sion channel objects forauserto select from, presenting 
them all at once would not only taice up significant 
screen space, but the visual context of the list (i.e., why 
Is the user looldng at this list in the first place) would be 
lost Thus, a method for allowing the user to browse 
through the available objects is required. 
[0075] The user interface of the present invention In- 
troduces an "object wheeT to provide a method for al- 
lowing a user to choose from a large group of objects. 
In the object wheel, a few immediately accessbie ob- 
jects are displayed in an object wheel on the display 
screen while the remaining objects in the group are not 
rendered on the screen. To access the objects not ren- 
(tered on the screen, the user "Spins" the object wheel 
to move the previously inaccessible objects into view on 
the display screen. 

[0076] Refening to Figure 12, a large number of fnjit 
objects are available for a user to choose from, yet only 
four fniit objects fit horizontally on the (fisplay screen 37 
at once. To provide the user with access to all the fnjIt 
objects available, the present invention puts the str^ of 
fruit ob^cts onto an "object wheeT 38 as displayed in 
the conceptual diagram of Figure 13. A few of the ob- 
jects on the object wheel 38 are immedlateiy accessible 
to a user on the display screen 37. To access the objects 
that are not displayed on the display screen 37, the user 



rotates the object wheel 38. As the object wheel 38 ro- 
tates, new objects are brought into view and other ob- 
jects disappear from view as illustrated in Figure 14. An 
operator uses the navigational gestures Point, Spin, and 

5 Pickup to manipulate the object wheel presented on the 
touch screen (Usptay 37. These three navigational ges- 
tures are explained in the following section. 
[0077] In order to enhance the user's feeling that a 
real wheel is being spun, object wheels obey the laws 

10 of physics. For example, the object wheels have mass 
and therefore inertia; object wheels spin faster as the 
user applies more "force" when spinning the wheel; ob- 
ject wheels resf>ond to friction since a spinning object 
wheel gradually stops spinning. 

IS [0078] While an object wheel is spinning and the ob- 
jects are moving across the screen, the cun^ent embod- 
iment of the user interface generates a "clicking" sound. 
The "clicking" sound generated by the user interface 
simulates the sound nuide by detents on a slot machine 

20 wheel whteh forces the wheel to stop and be aligned. 
Like a real wheel, the object wheels of the present in- 
vention always stop aligned, and never stop in between 
object choices. A user can stop a spinning object wheel 
at any time by Pointing to a moving object on the object 

25 wheel while the object wheel is spinning. 

[0079] An object wheel can be aligned vertically or 
horizontally on the display screen. When placed verti- 
cally, the object wheel can be spun both up and down. 
When placed horizontally, the object wheel can be spun 

30 both left and right 

User Interface Navigational Gestures 

[0080] Navigation and object manipulation in the user 

35 interface are performed using a screen pointing device, 
in the preferred embodiment, the hand-held display de- 
vk» 170 is constructed with a touch screen display 37 
and the pointing device used is a user's finger Altema- 
tively. the user interface may be Implenriented using any 

40 other type of screen pointing device such as a mouse, 
a trackball, or a joystick. This specification, however, will 
only describe the prefenBd embodiment that uses the 
user's finger as a pointing device on a touch screen. 
[0081 ] in the preferred embodiment of the present in- 

^ vention, the hand-hekJ display device 170 comprises, in 
part, a touch screen display 37 as iHustrated in Figure 
lb. The touch screen display 37 senses when a finger 
has been placed on the display screen and Mentifies 
where the finger has been placed. Users select objects 

50 on the touch screen and manipulate the objects using a 
set of six navigattonal gestures: Point, Select, Pan, Spin , 
Pickup, and Drop. Each navigatk>nal gesture used by 
the user Interface is explained individually t>eiow. (Since 
the nav^atk)nai gesture names are the same as com- 

55 mon words, this spedflcatton will capitalize aH referenc- 
es to the navigational gestures.) 
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Point 

[0082] Point is the most simple gesture in the user in- 
terface of the present invention. An operator uses the 
Point gesture to pick objects rendered on the display 5 
screen. For example, if the user Interface of the present 
invention requests the user to choose one of many ob- 
jects rendered on the display screen, the user Points to 
an object to choose the object. 
[0083] A user Points to an object by simpiy placing a 
finger on the object displayed on the touch screen. 
When a user has successfully Pointed to an object, the 
user interface generates a 'Selectable Objecf sound. 
The Selectable object sound provides feedbacic to the 
user indicating that the user has successfully Pointed to 
the object. 

Select 

[0084] The Select gesture is very similar to the Point 
gesture. An operator uses the Select gesture to have an 
object invoice a method to perf onn an function or action 
associated with theobject. For example, a Selected por- 
tal invoices a nr^ethod that moves the user to the Space 
associated with the portal and a Selected button Invoices 
a method that performs the action or function associated 
with the button. 

[0085] To Select an object, a user first Points to an 
object by touching the object with a finger. If the user 
continues to Point to the object for a predetermined nrtin- 
inujm period of time, the object becomes Selected. After 
a user Selects an object, the object "anticipates," gen- 
erates a sound for user feed>aclc, and then a method 
within tiie object is invoiced, if theobject is a button, then 
the mettiod invoked performs a function associated with 
the button, if the object is a portal, then the method in- 
voked will nrvove the user to the Space associated with 
that portal. 

[0088] Altematively, a user can Select an object on 
the display screen by qutekly Pointing to ttie object twice 
in successton. When a user Selects an object by quickly 
Pointing to tiie object twk», the object does not antici- 
pate, instead, the user interface on^ generates the ap- 
propriate feedback sound and then performs tiie action 
associated witti the object This provides the user a 
method of Selecting an object quickly. 

Pan 

[0087] To move around within a Space a user uses 
the Pan navii^tionai gesture. A user Pans though a 
Space by placing a finger on the background Image of 
a Space displayed on the touch display screen 37 and 
sIMing the finger in the desired direction of movennent 
The user interface of the present invention responds by 
generating a "Pan Active" sound and moving back- 
ground image of the Space and all the objects wittiin the 
Space In the opposite direction of the finger's move- 



ment. By moving the Space in the opposite direction, 
the user interface simulates movement of the users' 
point of view In the other direction. 
[0088] Rgures 4 and 5 illustrate the use of the Pan 
gesture. If a user places his finger on the t>ackground of 
Figure 4 and then slides the finger on the touch screen 
to tiie right, the view will con^espondingly pan to the right 
by moving the background image of the Space and ail 
the objects within ttie Space to the left. Figure 5 illus- 
trates the result of the Pan right operation where house 
48 has moved from tiie right side of the display screen 
over to the left side of tiie display screen. 
[0089] The background bnage for each Space In the 
present invention can be arbitrarily large. If the back- 
ground image of a Space has finite size, the user Inter- 
face of the present invention provides feedback that the 
furtiiest extent of the background image has been 
reached. When a user Pans In a direction all the way to 
tiie end of a background Image, the Pan gesture will not 
move the user any further In that direction. To Infomi the 
user of tills fact, the user interface generates an 'end of 
extenr sound Instead of the "Pan Active" sound when 
a user attempts to Pan past the end of a background 
innage. The "end of extent" sound sounds like a IhucT 
made when somettiing bumps Into a wall. The "end of 
extenr sound thereby inf omns the user that the user has 
hit ttie end of the background Infiage and can go no f ur- 
ttier. 

Spin 

[0090] An operator uses ttie Spin gesture to browse 
ttirough several objects placed on an object wheel. 
When a user Spins an object wheel, the visible objects 
on ttie object wheel move across ttie display screen and 
disappear off one end of the screen and previously in- 
visible objects move onto the display screen at ttie ottier 
end. 

[0091] A user Spins an object wheel by placing a fin- 
ger on the object wheel and ttien moving ttie finger 
across ttie display screen In a direction parallel to ttie 
object wheel's movement. Hgure 1 5 illustrates ttie Spin 
g^ture. in Figure 15, a user first places a finger on the 
object wheel (as shown by ttie asterisk 96). The user 
ttien sIkJes the finger across the object wheel (as shown 
by the anrow 97). This finger gesture causes ttie object 
wheel to Spin in ttie direction of ttie finger'^ movement 
The faster the user slides the finger across the object 
wheel on ttie display, ttie faster ttie object wheel spins. 
Once the object wheel is spinning, ttie objects on the 
object wheel appear at one end of an object wheel, 
move across ttie display screen, and ttien disappear off 
ttie ottier end of ttie object wheel sinnilarto figures whfeh 
appear on a sk>t machine wheel. 

Pickup 

[0092] The Pk^kup gesture is used to pk:kup an object 
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located within a Space or on an object wheel such that 
a user can carry the object around the "world" defined 
by the user interface of the present invention, in the 
present ennbodiment, different methods are used to 
Pickup an object from a Space or an object wheel. Each 
method will be explained IndMdually below. However, It 
will be appreciated that the same gesture could be used 
to pickup objects in both situations. 
[0093] The gesture used to Pk:kup an object from an 
object wheel Is illustrated in Figure 16a. Figure 16a U- 
lustrates an object wheel 91 with a number of fruit ob- 
jects to choose from. To PkHoip one of the fruit objects 
disposed on the object wheel 91 , a user first places a 
finger on the object (as designated by the asterisk 92) 
and nx>ves the finger in a direction perpendicular to the 
motion of the object wheel. The movement of the finger 
in a directkm perpencficular to the motbn of the object 
wheel Is Illustrated by arrow 93. The perpendfeular fin- 
ger motion "tears" the object off the object wheel such 
that the user can move the object. When the object 
comes loose from the object wheel, the user interface 
generates a "tearing" sound that Informs the user that 
the object has been removed from the object wheel. 
When an object has been picked up. the object casts a 
shadow onto the background image of the Space and 
moves the object atong with the user's finger 
[0094] The gesture used to Pickup an object from a 
Space is illustrated in Rgure 1 6b. Figure 1 6b illustrates 
a Space with an object in the center To Pk:kup the ob- 
ject, a user drcies the object with the finger as illustrated 
by arow 203 and then touches the object as illustrated 
by asterisk 201 . When the object comes loose from the 
Space, the user Interface generates the tearing sound 
that informs the user that the object has been pfcked up. 
After an object has been pk:ked up from a Space by a 
user, a ghost famage of the object remains at the object's 
original location. The ghost image of the object is drawn 
in desaturated colors indicatingto the userthat the ghost 
object cannot be Selected. 

[0095] After a user has picked up an object, either 
from an object wheel or a Space, the user Interface gen- 
erates a "carrying object" sound that infomns the user 
that an object Is being carried. The user can move the 
canried object within a Space, from Space to Space, or 
Drop the carried object onto another object. 

Drop 

[0096] The final gesture used to interact with the user 
Interface of the present Invention is Drop. After a user 
has used the Pickup gesture to pick up an object from 
a Space orfrom an object wheel, the user must continue 
to keep the finger pressed against the screen to cany 
the ot)ject. The user moves the canried object by sliding 
the finger across the touch screen display system. After 
moving the canted object to a desired iocatton, the user 
then uses the Drop gesture to place the object at the 
desired tocatlon. To drop the object the user lifts the fin- 



ger off the touch screen display. When the user Drops 
an object, the object falls into the background image of 
a Space as if pulled by gravity, and the object no longer 
casts a shadow. 

5 [0097] A user of the present invention uses the Drop 
gesture to perform a number of different functbns. The 
Drop gesture can be used to copy an object t>y pteking 
up the object and Dropping the object into a portal such 
that a copy of the object is created in the Space asso- 

10 dated with the portal. The Drop gesture can be used to 
move an object to a new location by pteking up the object 
and Dropping the object at the desired locatk>n. The 
Drop gesture can also t>e used to provkie a data object 
containing inf onnatbn to another object which uses the 

15 infomnatton to perfomn a function. Each different use of 
the Drop gesture will be explained below. 
[0098] To make a copy of an object using the Drop 
gesture, a user must first Pfekup the object and then 
Drop the object onto a portal. An object that has been 

^ picked up is carried over the portal by sliding the finger 
across the screen. Once the carried object is over the 
portal, the portal becomes h^hllghted such that it pro- 
vides feedback to the user signaling that the carried ob- 
ject is over the portal. In the present emtxKliment, the 

25 portal emits a green glow. To Drop the carried object into 
the portal, the user lifts the finger away from the touch 
screen display. The object is then displayed such that it 
passes through the portal with an appropriate anima- 
tion. During the animation, the CPU 30 generates a 

30 "sucked through portal" sound effect. When the object 
has completely disappeared through the portal, the 
ghost Image of the object drawn in desaturated colors 
In the present Space changes back into a brightly color- 
ed object This provMes visual feedback to the user that 

35 the original object remains in the present Space even 
though a copy of the object has been sent through the 
portal to another Space. 

[0099] Users can move objects either within a single 
Space orfrom one Space to another. To move an object 

40 within a single Space a user first uses the Pkdojp ges- 
ture to pick up an object, moves the object to the desired 
new location, and then Drops the object using the Drop 
gesture. After the object is Dropped, the ghost image of 
the object merges with the Dropped object at the new 

45 tocatlon to fonn a single brightly cotored image of the 
object at the new location. This infomisthe userthat the 
object has been moved to a new locatton and that a copy 
has not been made. 

[0100] Moving an object to another Space is accom- 
50 plished by canyingthe objectthrough a portal to another 
Space. To move an object to another Space, a usermust 
first Pk^kup the object and then cany the object over a 
portal. Once the carried object Is over the portal, the por- 
tal becomes highlighted such that it provkies feedback 
55 to the user signaling that the carried object is over the 
portal, and the portal emits a green gk>w. To move the 
canied object through the portal, the user hokis the ob- 
ject over the portal for a predeternilned minimum 
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amount of time. The portal eventually anticipates mov- 
ing the user and the canied object to the Space assod* 
ated with the portal. The ghost image of the canied ob- 
ject drawn In desaturated colors Is also moved along, 
with the user, and the carried object to the destination 
Space. When the user Drops the canied object by lifting 
his finger off of the touch screen display system, the car- 
ried object and the ghost Image merge Into a single 
brightly colored image of the object This provides feed- 
bade to the user that the object has been moved to the 
Space and has not been copied. 
[0101] The Drop gesture can also be used to provide 
infonnation to an object A user provides infomnatlon to 
an object by Dropping an object containing data (i.e. a 
data object) onto the object which will use that data. 
[0102] To provide information to an object, a user 
must first Pickup a data object that contains the neces- 
sary infonnation. The user then carries the data object 
over the object which can utilize the infomfiation. When 
the canied data object is over the object that can utilize 
the Infonnation, the object that can utilize the Informa- 
tion becomes highlighted if the carried data object is of 
the conrect dass. By becoming highlighted, the objed 
that can utilize the Information provides feedback to the 
user that the carried data object Is of the coned dass. 
To Drop the data object into the objed that can utilize 
the infonnation, the user lifts his finger used to cany the 
data obJed away from the touch screen . The data objed 
is then passed through and into the portal with appro- 
priate animation and sound effects. When the data ob- 
jed has completely disappeared Into the receiving ob- 
Jed whteh can use the lnfonnatk)n, the receiving obJed 
then perfonns the appropriate f undk>n on the data pro- 
vided. 

[01 03] For example, the Drop gesture can be used to 
provide television program Inf onnatton to the video cas- 
sette recorder object wh Ich is associated with a physk^ 
video cassette recorder, such that the video cassette re- 
corder programs itself to record the television program. 
To program the video cassette recorder, a user first uses 
the Pickup gesture to ptekup a data obJed containing 
television program informatk>n. The user then carries a 
television program data objed over the video cassette 
recorder objed The video cassette recorder obJed then 
modifies its appearance to indicate it is a "program video 
cassette recorder" mode. This Inf ornis the user that the 
video cassette recorder "understands" the infonnation 
that Is over it (I.e., the video cassette recorder will be 
programmed to record the television program If the user 
Drops the television program data object onto the video 
cassette recorder). 

[0104] When the user Drops the television program 
objed into the video cassette recorder, the video cas- 
sette recorder objed recognizes that the teievislon pro- 
gram data object contains infonnation tt can use. The 
video cassette recorder then extracts the tinte and chan- 
nel informatton from the television program data object 
such that the video cassette recorder objed programs 



itself to record the television program. 

A Tour of the Present Emijodiment in Operation 

5 [0105] To best convey the look and feel of the user 
interface of the present invention, a "tour" through a 
sample environment Is provided with reference to Fig- 
ures 3 - 10. The sample environment expired in the 
tour Is based upon the geographic map strudure of 
Spaces diagrammed In Figures 2b and 2c. The tour 
through the sample environment descrft>ed herein Is not 
Intended to limit the scope of the present Invention, but 
instead to provkie an example of one possible embodi- 
ment of the present invention. Due to the imitations of 
a written Spectfbatton in describing the present inven- 
tion's anvnated user interface, the reader is encouraged 
to view a vkteo tape entitled "Green Project User Inter- 
face," submitted by the appitoant concunently with the 
filing of the applk^on on whbh this patent Is based. 
[0106] Refening to Figure 3, upon first entering the 
user interface of the present inventton, the user interface 
initially positk>ns the user In an outer-space Space 99. 
This Space is illustrated in the geographte map stnJdure 
of Spaces dlagranmed In ngure 2b. Although, the user 
is initially placed in the outer-space Space 99, the user 
Interface will later start the user at the last Space the 
user was In when the user quit using the hand-heki dis- 
play device. Referring again to Figure 3, at the outer- 
space Space 99, the user interface renders a brightly 
cotored Earth portal 100 against a dark outer-space 
background Image on the display screen. 
[0107] Soon after entering the user Interface, an out- 
line of a hand 31 appears on the display screen to teach 
the user how to navigate around the user interface. Ihe 
hand 31 that appears on the display screen is the hand 
of the Agent The Agents hand 31 extends a finger to 
touch the screen and then sikies the finger across the 
screen causing the Space and objects on the screen dis- 
play to Pan and the user interface to emit the "Panning 
active" sound. Refening to Figure 3, the outBne hand 
31 of the Agent Is shown moving across the screen. By 
touching the display screen and moving a nnger across 
the display screen, the Agenfs hand 31 derDonstrates 
to a new user how to use the Pan gesture to move 
around the view of a Space, 
pioq After demonstrating how to use the Pan ges- 
ture to move around in a Space, the hand31 ofthe Agent 
next demonstrates how to use the Point and Seled ges- 
tures. The Agent demonstrates how to use the Point 
gesture to Point to an objed by pladng a finger on the 
Earth portal 100. After the Agent Points to the Earth por- 
tal 100, the user interface emits the "Selectabie ObJecT 
sound to provide feedback whk^h Informs the user that 
the Earth portal 100 is a Selectable Object The Agent 
then demonstrates how to use the Seled gesture to Se- 
lect an object by continuing to Point to the Earth portal 
100. Aflerthe Agenfs hand 31 Points to the Earth portal 
100 for the predetermined minimum amount of time, the 
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Earth portal 1 00 antic^tes and the CPU generates the 
"noove through portaf* sound. The Earth portal 100 an- 
ticfpates by expanding rapidly such that the Earth portal 
100 appears to move doser to the user. The screen then 
fades and a new background Image appears displaying 
the Earth's surface Space 32 associated with the Earth 
portal 100, as illustrated in Figure 4. 
[0109] At the Earth*^ surface Space 32, the Agent por- 
tal 41 appears to "jump" onto the touch screen display 
37 and moves to the lower left comer of the display 
screen. The Agent 41 win always be accessible to the 
user because the agent appears in every Space, except 
the Agent* s Space. If desired, the user can move the 
Agent from the lower left comer to a different location 
on the display screen. This movement of the Agent will 
change the Agent* s location in every Space. 
[01 1 0] Referring to Figure 4, in the lower right comer 
of the display is the Waybadc portal 45. The Wayt)aclc 
portal 45 displays a framed snapshot of the outer-space 
Space 99 from Figure 3 which the user Just came from. 
If the user Selects the Wayt>ack portal 45, the user in- 
tertace of the present invention moves the user bacic 
along the traveled path to the Space iNustrated within 
the Waybadc portal 45. For example, If the user desires 
to return to the outer-space Space 99 displayed in the 
Waybadc portal 45, the user simply Selects the Way- 
back portal 45. 

[0111] The Earth's surface Space 32 depicted in Fig- 
ure 4 is comprised of a number of houses and commu- 
nity buildings such as stores, banks, schools, etc. Each 
building in the Earth's surface Space 32 represents a 
portal that can be Selected by the user to move the user 
to a Space assodated with the portal. For example, in 
the Earth's surface Space depicted in Figure 4, the first 
house 47 and the second house 48 are portals whteh 
can be Selected k>y the user. If the user Selects either 
the first house 47 or the second house 48, the user will 
be nooved to an assodated house Space. 
[0112] While within the Earth's surface Space depkn- 
ed in Rgure 4, the user can move around in the Space 
by using the Pan gesture on the touch screen as dem- 
onstrated by the Agent* s hand 31 in Figure 3. To Pan, 
the user simply places a finger on the screen and moves 
the finger in the direction the user wishes to move. The 
user interface responds to the user's finger motion by 
generating the "Panning Active" sound and moving the 
background image and the objects on ttie screen in the 
opposite direction of the finger's movement, thereby 
bringing other objects into the view. 
[0113] For example, if a user viewing the Earth's sur- 
face Space 32 as depk:ted In Figure 4 wishes to move 
to the right, the user shiply places a finger onto a back- 
ground area of the Earth's surface Space 32 and slides 
the finger to the right. The user interface reciprocally re- 
sponds to the user's finger motton l>y moving the 
Space's background image and the objects in the Space 
to the left. Figure 5 depicts the view of the Earth's sur- 
face Space 32 after the user has placed a finger onto a 



background area of Figure 4 and slid the finger to the 
right A first house portal 47 displayed In Figure 4 has 
moved out of the view and is not visible In Figure 5. A 
second house portal 48 displayed on the right side of 
5 the view in Figure 4 has moved to the left side of the . 
view in Rgure 5 and now is only partially visible. A bank 
portal 49 that was not visible in Figure 4 has moved Into 
view from the right as Olustrated In Figure 5. The Agent 
41 and Wayback portal 45 remain in the view in the same 
tower left and lower right positions respectively. 
[0114] When the user Selects any portal from the 
Earth's surfece Space 32 as depicted in Figures 4 and 
5, the Selected portal anticipates to provkle feedback to 
the user and the user interface moves the user to a 
Space assodated with the Selected portal. For exanv 
pie, when a user at the Earth's surface Space 32 depk^- 
ed In Figure 4 Selects the first house portal 47, the 
house portal 47 anttolpates and then the user interface 
nrioves the user to a house Space associated with the 
house portal 47. Figure 6 Illustrates a house Space 50 
assodated with the house portal 47 of Rgure 4. 
[0115] The house Space 50 displayed in Rgure 6 
consists of a house with the outer walls removed such 
that the Indh^al rooms InskJe the house ars visible. 
All of the roonr» inside the house, such as the living room 
54, bedroom 53, bathroom 55, attto 52. and kitchen 56 
depicted in Figure 6, are portals that can be Selected 
by the user. As illustrated in Figure 6, the Wayback por- 
tal 45 contains a miniature snapshot of the Earth's sur- 
face Space 32, as it appeared when the user left the 
prevtous Space. Thus the Wayback portal 45 deptoted 
in Rgure 6 not only contains a miniature view of the 
Earth's surface Space, but a view of that portton of the 
Earth's surface Space 32 that was in view when the user 
Selected the house portal 47 of Rgure 4. To move back 
up to the Earth's surface Space of Rgure 4, the user 
Selects the Wayback portal 45 that contains a snapshot 
bnage of the Earth's surface Space. 
[0116] Refenlng to Figurs 6, if the user Selects the 
Ihring room portal 54, the Oving room portal 54 anttoi- 
pates, the "move through portar sound Is generated, 
and the user Interface moves the user to a Hving room 
Space assodated with the IhrIng room portal 54. Refer- 
ring to Figure 7, an embodiment of a living room Space 
60 is deptoted. Within the living room Space SO of Rgure 
7 is a television 73, a video cassette recorder 75. and a 
television program schedule book 71. The television 73. 
vkieo cassette recorder 75, and television program 
schedule book 71 are each portals that can be Selected 
by the user. The television 73 and the video cassette 
recorder 75 portals are associated with a real televiston 
and video cassette recorder respedh^. 
[0117] When a user Selects the televiston portal 73 
displayed within the living room Space 60 of Rgure 7. 
the tetoviston portal 73 antidpates and the user Iriterface 
displays a grsphtoal user interface assodated with the 
real television. Rgure 8 deptots the television control 
Space 64 which comprises the televiston's graph toal us- 
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er interface in the present embodiment. The objects 
available in the television Space 64 ^ude a green "on' 
button 61 , a red "off" button 62, and several television 
channel buttons 66 - 69. The television channel buttons 
66 - 69 are on a horizontally alined object wheel 63 
such that a user can Spin the object wheel 63 left or right 
in order to reveal additional television channel buttons 
that are not cun-ently displayed on the display screen. 
[01 16] The on button 61 . off button 62, and television 
channel button 66 - 69 objects available in the television 
Space 64 are not portals that give access to other Spac- 
es in the Space map of Figure 1 b. Instead, the on button 
61 , off button 62, and television channel buttons 66 - 69 
are buttons that are associated with functions. For ex- 
ample, the green "on" button 61 and red "off" button 62 
are associated with the functions "turn the television on" 
and "turn the television off" respectively. When a user 
Selects either the on or off button, the button anticipates 
to provide feedback to the user that the button was Se- 
lected. The hand-held display device then transmits the 
appropriate convnands that turn the television on or off. 
Similarty, If the user Selects any of the television channel 
buttons 66 - 69, the television channel button anticipates 
and the hand-held display device transmits the appro- 
priate commands to the real television thereby tuning 
the television to the Selected channel. The process of 
transmitting comn^ds to the physical televbion set Is 
explained in detail in the following section. 
[0119] Referring back to Rgure 7, ff a user Selects 
the television program schedule book portal 71 , the tel- 
evision program schedule book portal 71 anttelpates, 
and the user Interface moves the user to the television 
program schedule Space. Figure 9 depicts the televi- 
sion program schedule book Space of the present em- 
bodiment In the televiskH) program schedule book 
Space of Figure 9, the user Is presented with a two di- 
mensional anay of televisk)n program listings. The an^ 
organizes the televiston program listings vertically b)f 
television channel and horizontally by time. Each tele- 
viston program ysting Is aportal that can be Select by 
the user. Each television program listing can also be car- 
ried by the user and Dropped onto the video cassette 
recorder portal such that the hand-heki display devk^e 
progranis the video cassette recorder to record the tel- 
evision program. 

[01 20] The television program schedule book Space 
of Figure 9 is actually organized as a pair of connected 
vertk:ally oriented objectwheeis. Referring to Hgure 11, 
a conceptual diagram of the teieviston program sched- 
ule book Space is illustrated. The televlsk>n channels of 
Figure 9 are placed on a first object wheel 72 as illus- 
trated in Rgure 11 . The television program listings are 
placed a second object wheel 85. The two object wheels 
72 and 85 are connected such that if the user Spins ei- 
ther object wheel, the other object wheel will spin at the 
same rate such that the television channels and televi- 
sion program listings are always aligned. 
[0121] Each television program listing on the televi- 



sion program schedule object wheel 85 Is a portal. When 
a user Selects one of ttte television program listing por- 
tals, the television program listing portal anttefpates and 
the user Interface moves the user to a television pro- 

5 gram Infonmatlon Space. Figure 1 0 illustrates a typical 
television program informatton Space. The television 
program Infonmatlon Space Includes a dock 81 , a date 
82 and televiston channel 83 whteh Inform the userwhen 
the television program Is on and what channel the tele- 

10 vision program is on. The television program inf omnatton 
Space also includes a descr^on of the televiston pro- 
gram as illustrated in Figure 10. 



IB 



The Agent 



10122] The Agent Is an anthropomorphic character 
designed to help a user of the present inventton in a 
number of ways. The Agent assists users by providing 
help, providing state infomnation, alerting the userto cer- 
» tain events, and acdng as a repository. Each functton 
perfomned by the Agent wifl be explained individually. 

Help 

^ [0123] As previously described In the tour of the 
present enrolment, with reference to Figure 3. when 
a user first enters the user Interface of the present in- 
vention, the Agent demonstrates how to use the Point 
gesture and the Select gesture. Furthennore, the Agent 

50 demonstrates how to use the Pan gesture to move 
around within a Space. The Agent can also demonstrate 
how to use other features of the user Interface as nec- 
essary. 

[0124] To further akJ users, the Agent carries a help 
35 manual in Its briefcase that Is available to the user at 
any time. To access the help manual, the user simply 
Selects the Agent portal. The Agent antlctpates by pro- 
vkSing an animation of the Agent opening up Its brief- 
case. The user then Selects the help manual from within 
40 the Agents briefcase. 

Stale and Mode Information 

[01 25] The Agent Is also used to express the current 
45 State or mode of the user interface. To express the cur- 
rent state or mode the Agent changes its ctothing, 
changes Its expresston, or changes the props It Is car- 
rying. 

[0126] Forexampie, iftheuserlnterface of the present 
50 invention Is placed into a diagnostic mode, the clothing 
that the Agent Is wearing Is changed to convey this fact. 
Referring to Figure 17a, the Agent is shown In a doctor's 
unifomfi to symbolize that the user Interface Is In a diag- 
nostic mode. 

55 [0127] Figure 17b provkies another example of how 
the Agenfs appearance can convey stale Information. 
When the user Is in a writing state, the Agent is given a 
pencil to symbolize that the user Interface is In a writing 
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state. 
Alerts 

[0128] While a user Is using the hand-held display de- 
vice of the present invention, certain events may occur 
that require the user to be Infonned Innmedlatety. Such 
events are usually referred to as alerts. For example, if 
the hand-held display device is coupled to an electronic 
messaging service, the hand-held display device should 
alert the user when a message has been received that 
Is addressed to the user. 

[0129] In the user interface of the present invention, 
the Agent Is used to obtain the user's attention in order 
to inform the user about any alerts. To get the user's 
attention the Agent waves its hand. Additionally, If the 
alert is very Important, the Agent may wave both amns 
or produce sounds to further attract the user's attention. 
[0130] For example, when an event occurs that re- 
quires the user's attention, the Agent begins waving as 
illustrated in Rgure 17c. To explain what caused the 
alert, the Agent* s appearance may be further altered as 
demonstrated in the previous section in order to convey 
Infonnatlon about the specific alert For example, when 
an urgent message arrives, the Agent can wave with 
one hand while holding a telegram In the other hand. 
[0131] Since alerts are generally more importantthan 
state or mode infonnation, alerts will take precedence 
over state or mode information. Therefore, the image 
of the Agent Is currently being used to convey state or 
mode Information when an Important event occurs, the 
image of the Agent will change to get the user's attention 
and Inform the user at)out the alert For example, if the 
Agent is holding a pencil while in writing mode, as illus- 
trated in Rgure 17b, when an urgent nrtessage anives, 
the Agent will bnmediately change to the waving anima- 
tion as illustrated in Hgure 1 7c to infomn the user about 
the message. 

Remote Device Contiol 

[01 32] The hand-held display device of the present In- 
vention Is designed to control remote devices. Accord- 
ingly, the user interface defined in the previous section 
for the hand-held display device is designed to be an 
Ideal graphical user Interface for contn>illng remote de- 
vices. For example, to control a device In the user's liv- 
ing room, a user first moves to the living room Space 60 
of the user Interface as illustrated In Rgure 7. The user 
then Selects an object conresponding to a device that 
the user wishes to control. For example. If the user de- 
sires to control a television, the user Selects the televi- 
sion portal 73 associated with a real television. When 
the user Selects the television portal 73. the television 
portal 73 anticipates and then displays a graphical user 
interface for controlling the physical television in the us- 
er's living room. Figure 8 Blustrates one possible em- 
bodiment of a set of television controls for controlling a 



television. The user then controls the television using 
the set of television controls displayed on the hand-held 
display device. 

[01 33] In order for a remote device to be controlled by 
5 the hand-held display device, the hand-held display de- 
vice must have certain information about the remote de- 
vice. Specifically, the hand-held display device must be 
provided with an object-oriented programming type of 
object that is associated with the remote device. The ob- 
10 ject comprises a plurality of methods Including a method 
for displaying the object on the display screen of the 
hand-held display device and a method for displaying a 
graphical user interface for the remote device. 
[01 34] To simplify the operation of the present Inven- 
ts tion, each compatbie remote device automatically 
transmits the required object-oriented programming 
type of object to the hand-held display device. When a 
remote device transmits the object to the hand-held dis- 
play device, the hand-held display device Invoices the 
method for displaying the object on the display screen. 
The user can then move the object to an appropriate 
Space, if the userSeiects the object, the Selected object 
anticipates and then the nf)ethod for displaying the 
graphical user interface is invoiced. The method displays 
the remote device's graphical user interface on the 
hand-held display device's touch screen display. The 
user can then control the remote device kyy interacting 
with the remote device's graphical user interface dis- 
played on the touch screen display. 
[0135] Thehand-heiddlsplaydeviceofthepresentin- 
vention automatically obtains a graphicai user Interface 
from two types of compatS>le remote devices. The first 
type of compatft)ie remote devtee is the Intelligent re- 
mote device with a dedicated computer control system. 
The second type of compatible remote device is a sim- 
ple remote device that can only respond to simple coded 
commands. 

[0136] The hand-held display device Interacts differ- 
ently with the two types of compatible remote devices. 
The hand-held display device communicates with Intel- 
ligent remote devices using peer-to-peer communica- 
tion with remote method invocation for remote device 
control. The hand-held display device communicates 
with simple remote devices using only simple control 
codes. However, from the user's perspective the differ- 
ences between the Intelligent rennote devices and the 
simple remote devices cannot be seen, and the user's 
interaction with the graphicai user interface is identical 
for both types of remote devices. The implementation 
detail of whether a user's actions cause a remote meth- 
od invocation, or transmission of control codes tiirough 
some communication mechanism is hidden from the us- 
er. 

pi 37] This section wHi explain how the two types of 
connpatft>le remote devices transmit infonmatioh to the 
hand-held display device. Furthermore, a third type of 
remote device wID explained. The third type of device 
consists of any electrical device that can be controlled 
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by coded infrared signals. 

[0138] Within this section, reference will be niade to 
object-oriented progranvrting type objects. Each object- 
oriented programming type of object contains data and 
methods that can be Invoked to perform functions. To s 
simplify the tamrtlnology used, the temn "program obJecT 
will be used when referring to an object-oriented pro- 
gramming type of object. To eliminate confusion, the ob- 
jects displayed on the touch screen display in the user 
interface disclosed in the previous user interface will be 
refenred to as 'display objeds" In this section. 

Peer to Peer Communication With Remote Method 
invocation 

[0139] in the present invention, a remote device is de- 
fined as an "Intettlgent remote device" if that remote de- 
vice contains a microprocessor, a communication sys- 
tem, and a sufficient anrK)unt of memory such that It can 
invoke methods In a program object. Figure 1c illus- 
trates, in block diagram fomn, the components In an in- 
telligent remote devk». The hand-heM display devk» of 
the present Inventton communteates with intelligent re- 
mote devices In a peer-to-peer fashion. 
[01 40] The intelligent remote devfee has an associat- 
ed graphical user Interface comprising touch screen 
control elements such as dials, switches, sliders, etc. 
The intelHgent remote device's graphical user interface 
is defined within a user interface program object 303 
stored In the intell^nt remote devk»'8 memory 404. 
When an Intelligent remote devtee is first tumed on. the 
intelligent remote devtee automatk^lly transmits its user 
interface program object 303 to the hand-hekl display 
devk». The hand-held display devfee can then display 
the user Interface for the intelligent remote device as 
defined by the user Interface program object. 
[0141] When a user interacts with the intelligent re- 
mote device's graphk^ user interface as displayed on 
the hand-heki display devk», the hand-hekJ (fisplay de- 
vice invokes m^ods In a devk:e drWer program object 
301 within the intelligent remote to control the remote 
devtee. Thus the software that actually "controls' an In- 
telligent remote devk:e Is tocated within the Intelligent 
remote devk^e Itself. 

[0142] The procedure <rf transmitting the user Inter- 
face object 303 from an Intelligent remote devtee to the 
hand-held display devk» win be disclosed with refer- 
ence to Figures 18a - 18d, 19a • 19d and the fk>wchart 
In Rgure 20. Hgures 18a - 18d illustrate, on a high lev- 
el, how the Intelligent remote devfce 150 and the hand- 
held display devbe 170 of the present Inventton com- 
municate with each other. Figures 19a ■ 19d illustrate, 
in conceptual form, how the user interface program ob- 
ject 303 in the Intelligent remote devtee 150 is transmit- 
ted to the hand-held display device 170. 
[01 43] Refenring to step 1 01 of Figure 20, when a new 
Intelligent remote devk:e 150, such as the intelligent 
thennostat control device in Figure 18a, Is first tumed 



on. the new intelligent remote device 150 broadcasts 
across the communicattons network 160 that It has a 
user interface program object to export. As set forth In 
the hardware section, the communication on the com- 
munk^ations network 180 can take place using radk> 
waves, low power, cellular, Infrared signals, or electrical 
signals on existing power lines In the case of an Eche- 
k>n9 base LONO system. In step 102, a hand-held dis- 
play device 170 receh/es the interface export message. 
The hand-heki display device 170 signals back across 
the communk^tions network 160 to the new intelligent 
remote devtee 1 50 requesting the new Intelligent devtee 
150 to begin transnrmting the user Interface object 303. 
[0144] The Intelligent remote devtee 150 then trans- 
mits the user interface object 303 to the display d6vk» 
170 as specified In step 103 of Figure 20 and illustrated 
In Figure 18a. Refening to Figure 19a, the Intelligent 
remote devtee 1 50 also transmits a pointer (as indk»ted 
by dotted lines) to a devk» driver program object 301 
within the Intelligent remote device 150. The user Inter- 
face program object 303 defines how the user interface 
for the intelligent remote devk» 150 shouki appear on 
a display screen. The user interface program object 303 
invokes methods In a device driver program object 301 
to handle user interface interacttons. The device driver 
program object 301 contains methods that control the 
actual intelligent remote devtee 150. Referring to Figure 
1 9b, the display devtee 170 now has a copy of the intel- 
ligent device's user interface program object 313 and a 
device drWer program object 311 pointer. 
[01 45] After receiving the user interface program ob- 
ject 313. the hand-heW display devtee 170 Invokes a 
m^od in the user Interface program object 313 that 
renders a display object in the current Space as stated 
in step 104. The new intelligent devtee 1 50 also registers 
with a remote name server connected to the communi- 
cations network 160. 

[0146] When a user Selects the display object asso- 
ciated with the new Intelligent renrK>te devtee 150, the 
hand-heU cfisplay device 170 invokes a method in the 
user interface program object 313 as stated in step 105 
(the object with an Invoked method Is shown in bold). 
The nnethod Invoked In the user interface program ob- 
ject 313 displays the remote intelligent devtee's graphi- 
cal user Interface on the display screen 171 of the dis- 
play devtee 170 as Illustrated In Figure 18b. A user can 
interactwith the remote intelligent devtee'a graphteal us- 
er interface displayed on the display devtee 170 as stat- 
ed in step 108. 

[0147] The user controls the intelligent remote device 
150 by Interacting with the graphteal user interface dis- 
played on the hand-held display devtee 170. To resolve 
the user's interacttens. the user Interface program ob- 
ject 313 requires control methods In the intelligent re- 
nrK>te device's device driver program object 301 . To ac- 
cess the control methods in the Intelligent remote de- 
vice's devtee driver program object 301 , the hand-heki 
displc^ devtee 170 uses the devtee drWer program ob- 
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Ject pointer 31 1 . The device driver program object point- 
er 31 1 invoices the methods in the real device driver pro- 
gram object 301 . The intelligent remote device's device 
driver program object 301 Is invoiced t>y sending a mes- 
sage over the communication networic 1 60 as illustrated 
in Figures 18e and 19c (again, the object with an in- 
voked method Is shown in bold). The methods invoked 
In the Intelligent remote devtee'e devk» driver program 
object 301 control the intelRgent device 150 as stated In 
step 108. 

[01 48] Finally, in step 1 09, any return values or feed- 
back generated by the device driver program object 301 
In intelligent remote device 150 are sent back to the 
hand-hetd display device 1 70 over the communications 
network 180 as illustrated In Figures 1 6d and 19d. The 
user interface program object 313 changes the graphi- 
cal user interface on the display screen 1 71 In response 
to the return values or feedback received from device 
driver program object 301 In the Intelligent remote de- 
vice 150. 

[0149] A robust communk:atk>n protocol across the 
communk:ations network 160 is required to communi- 
cate remote method invocations between the hand-heki 
display device 170 and the InteHigent remote device 
150. Both the hand-hekJ display devtee 170 requesting 
the method invocation and the intelligent remote devtee 
150 that executes the method must be abie to handle a 
broken communication path. Therefore, both the hand- 
held display device 170 and the intelligent remote de- 
vice 150 have a dedicated rennote procedure call (RPC) 
manager that handles the remote method invocatton. 
Referring to Figures 19a - 19d. the remote procedure 
call manager 309 transmits the remote method invoca- 
tk>n request for the hand-held display devtee 170. On 
the other side, a server remote procedure call manager 
307 processes the received remote method Invocation 
in an intelligent remote device 150. 
[0150] Figures 21a and 21b illustrate state diagrams 
used to control the remote procedure call (RPC) man- 
agers that handle the remote method Invocation on both 
the hand-hekl display device 170 skte and the remote 
devtee side. Taken together, state diagrams In Figures 
21a and 21b disctese a handshake used between the 
hand-held display devtee 170 and an intelligent remote 
devtee 150 to ensure reliable communteations. 
[0151] Figure 21a illustrates a state machine for the 
remote procedure call manager 309 in the hand-held 
display device 1 70 that handles the display devtee's side 
of a remote method invocation, in the present inventten, 
the user Interface program object 313 in the hand-held 
display devte0l7O requests remote method invocations 
to the device driver program objects 301 tocated In re- 
mote intelligent devtees. Refening to Figure 21a, when 
a user interface object requests a remote method invo- 
catten,the hand-hetel display devtee's RPCmanager en- 
ters the "ready to transmif state 403. The display de- 
vtee'b RPC manager then sets an error countdown timer 
and transmits a packet containing a message with a 



method selector and associated data. After transmitting 
the packet, the display devtee's RPC manager enters a 
"waiting for reply" state 405. If the proper reply message 
is sent back, the method invocatten is complete and the 
5 reply message Is analyzed at state 407. 

[0152] Remote method invocations do not always 
complete so easily, if the device driver program object 
301 in the inteUigent remote devtee150 needs more time 
to complete the nrtethod, it wiH send back a message 
10 requesting more X\me. The display devtee's RPC man- 
ager responds by moving to state 409 where it increases 
the timeout value. If the display devtee's RPC manager 
does not receive a reply by the time the enror countdown 
timer expires, the display devtee's RPC manager moves 
IS to an "error recover/ state 41 1 . If the error countdown 
timer expires tess than a predetermined number of 
times, the display device's RPC manager r^ms to the 
"ready to transmlT state 403. If the enor countdown tim- 
er reaches a predetemilned Hnnit, the display devtee's 
RPC manager abandons the rennote method lnvocatk>n 
and enters a failed state 413. The user Is then usually 
infornoed of the communteations probtem. 
[01 53] Hgure 21 b Illustrates a state machine for a re- 
mote procedure call manager 307 in an intelligent re- 
mote devtee that handles the remc^e devtee's side of a 
remote method invocation. The remote devtee's RPC 
manager 307 starts In a "waiting to receive" state 451 . 
When a packet is received, the remote devtee's RPC 
manager analyzes the packet at state 453. If tiie re- 
ceived packet contains a new message requesting a 
method Invocation, the remote devtee's RPC manager 
gives the packet to an object that perfonm tfie method 
of the request at state 455. The remote device's RPC 
manager then transmits a result and returns to the wait- 
ing state 451. 

[0154] If the received packet contains a message tiiat 
ihe remote devtee has already seen, the RPC manager 
of the Intelligent remote devtee moves to state 457 
where the intelligent renriote device detemilnes whether 
the method invocation request was already completed 
or whether it Is still work^g on the request. If tiie Intelli- 
gent remote devtee has already completed the request, 
ttie sut>system retransmits the results at state 459 and 
returns to tiie waiting state 451 . Otiierwise, If tiie Intel- 
ligent remote devtee Is still working on the method invo- 
cation, ttie remote devtee's RPC manager moves to 
state 461 where the remote devtee'b RPC manager re- 
quests more time to complete the invoked method and 
ttien returns to the waiting state 451 . 
[0155] Figure 21c illustrates the sequence of a re- 
mote method invocatten where ttiere are no communi- 
cation problems. At step 501, a user interface object in 
ttie hand-hekl display devtee requests a remote method 
Invocation. Therefore, the user interface object sends a 
message witii a metiiod selector and argument to the 
display device's RPC manager. The display devtee's 
RPC manager packs ttie message recehred Into a pack- 
et at step 502 and transmits ttie packet across ttie com- 
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munications network at step 503. 
[01 56] The intelligont remote device's RPC manager 
receives the transmitted packet at step 504. The remote 
devtee's RPC manager unpacks the message from the 
packet at step 505 and directs the packet to the devk^e 5 
driver program object within the intelligent remote de- 
vice. The device driver program object Invokes the 
method specified by the method selector In the message 
at step 506. The device driver program object generates 
a retum message that is passed back to the remote de- 
vtee's RPC manager. At step 507, the remote devk»*s 
RPC manager packs the retum message Into a reply 
tnjffer. The remote devfee's RPC manager sends the re- 
ply buffer at step 508. 

[0157] The display devk:e*s RPC manager receives 
the reply buffer at step 509 and unpacks the reply buffer 
to extract a retum value at step 510. The display de- 
vice's RPC manager delivers the retum value to the re- 
questing user interface program object that invoked the 
method in the devk» driver program object Finally, the 
user interface program object analyzes the retum value 
at step 511. 

[0158] As illustrated, thepresent inventk>n can control 
an intelligent remote device by remotely Invoking meth- 
ods In a device driver program object k)cated within the 
intelligent rerTK>te devtoe. In addition, it will be appreci- 
ated by the reader that the remote devk» may also re- 
motely invoke methods in the display devk». For exam- 
ple, the devtee driver object in a remote devk» may in- 
voke a method In a user Interface object In the display 
devk>e to display a message. 
[0159] Many renrK>te devbes are not sophistk^ted 
enough to have internal control methods that can be in- 
voked by the display devbe. Therefore, the present In- 
vention provkJes other methods for controlling simple 
devices. 

Sbnpie Device Control 

[01 80] In the present inventton, a remote devk:e is de- 
fined as a 'simple device' the remote devk:e does not 
have a microprocessor or the remote devtee contains 
such a primitive processor that the display device can 
not invoke control methods within the remote device. 
Rgure Id IBustrates. In block diagram form, the compo- 
nents in a simple remote devtee. To control a sfanpie de- 
vk». the hand-held display devk:e of the present inven- 
tk>n transmits simple predefined control codes instead 
of Invokingcontrol methods within the simple remote de- 
vice. Therefore, to control a simple remote device, the 
hand-held display devtee must otytain both a user inter- 
face program object and a device driver program object 
from the simple remote devtee. 
[0161] An example of obtaining user interface pro- 
gram object and a devbe driver program object from a 
simple remote devk» is given with reference to Figures 
22a - 22d, 23a - 23d and the ftowchart in Figure 24. 
ngures 22a - 22d illustrate on a high level how the re- 



mote simple device 155 and the hand-held display de- 
vice 170 of the present invention communicate with 
each other. Figures 23a - 23d illustrate, in conceptual 
f omn, how the user interface program object 353 and the 
devk:e driver program object 351 of the remote external 
devbe 1 50 are transported the hand-hekl display device 
170 and used. 

[0162] Refening to Figure 22a, a thermostat imple- 
nrtented as a sbnpie remote devtee 155 is niustmted. As 
specified by step 124 of Figure 24. the simple remote 
devfee 1 55 of Figure 22a transmits both a user Interface 
program object and a devk» driver program object to 
the hand-heM display devtee 170. Illustrated conceptu- 
ally in Rgure 23a. the simple devtee 155 transmits the 
user interface program object 353 and the devfc» driver 
program ob^ 351. The transmission of the program 
objects is done using some comnruj nicatk>ns means 1 62 
that may be a communtoattons network or any other 
means of transferring the Infomnation into the display de- 
vteelTO. 

[01 63] The hand-held display devtee 170 receives the 
copy of the user Interface program object 363 that de- 
scribes the simple devk^e's graphteal user interface. The 
hand-hekl display devk» 170 then invokes a method In 
the userlnterface program object 363 that rendersadis- 
ptay object within a Space shown on the hand-held dis- 
play devk» 170. When a user Selects the display object 
associated with the simple remote devbe 155, the dis- 
play devtee 170 Invokes a method within the user inter- 
face program object 363 as stated in step 125 of Rgure 
24 and illustrated in Rgure 22b. The method Invoked 
in the user interface program object 363 displays a 
graphk^al user interface associated with the simple re- 
mote device on the display screen 171 of the display 
device 170. The user can then interact with sfanpie re- 
mote device's graphical user Interface on display devfee 
170 as stated In step 127. 

[0164] Theuser's interac^onswiththeslmpledevtee's 
graphical user interface are analyzed by the simple de- 
vk»'s devk;e driver program object 361 within the dis- 
play device 150. The device driver program object 361 
resolves the user's interacttons Into control codes that 
the simple remote devk» 155 will respond to. As illus- 
trated in Figure 23c, the devtee driver program object 
361 located within the display devtee 170 analyzes the 
user's actions. 

[0165] Refening to Rgures 22c and 23c, the hand- 
heki display devtee 170 transmits the control codes to 
the simple remote device 155. The simple remote de- 
vk» 155 changes In response to the control codes re- 
ceived as stated In step 130. Finally, if the simple remote 
devtee 155 and the display devtoe 170 are coupled to- 
gether by a communk^ons network, the simple devtee 
155 can transmit any retum values or feedback gener- 
ated by the simple remote device 155 as lOustifated in 
Rgure 22d. 
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Conventional Device Control 

[01 66] Existing remote devices that are not designed 
for the hand>held display device of the present invention 
do not have a user interface program object to transmit 
to the hand-held display device. For exampie. Rgure 
1e illustrates a ^ical existing renwte device that can 
only respond to coded command signals such as infra- 
red signals. In such cases, the hand-held display device 
must be manually infomned about the remote device. 
The hand-held display device then uses a generic user 
Interface program object for the type of reniu>te device. 
[0167] For example, if a user has a conventional tel- 
evision that is controlled by infrared signals, the user 
enters a model number of the television to inform the 
hand-held display device about the conventionai televi- 
sion. The hand-held display device then accesses a da- 
tabase to obtain a list of the control codes that the con- 
ventional television will understands. A television dis- 
play object is then created to control the conventional 
television. If the user selects television display object, 
the hand-held display device will Invoke a method that 
displays a generic television user Interface. When the 
user Interacts with the generic television user Interface, 
the display device will transmit the proper infrared con- 
trol codes to control the television. 

Suwmary 

[0168] The foregoing has described an Intuitive 
graphical user Interface based upon a geogregDhic map 
structure and a system for controlling remote extemal 
electronic devices. In the user interface of the present 
invention, each Space of the geographic map stnicture 
Is rendered on a touch screen display as a Space. Within 
each Space are colored icons called "objects' that can 
be Selected and manipulated by the user. Certain ob- 
jects, referred to as portals, transport the user from one 
Space to another Space when Selected. Other objects, 
referred to as buttons, perform associated actions or 
functions when Selected. The user Interfece Is dis- 
played on a hand-held display device that can control 
renrK)te electronic devices. Each remote electronic de- 
vice transmits a user interface program object that de- 
fines a graphical user interface to the display device. 
The hand-held display device displays the graphical us- 
er Interface associated with the remote device when a 
user selects an icon associated with the remote device 
on the display device's touch screen display. It Is con- 
templated that changes and modifications may be made 
by one of ordinary skill in the art, to the materials and 
anrangements of elements of the present Invention with- 
out departing from the scope of the inventH>n. 



Claims 

1. A method of provMIng a graphical user interface on 
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a computer system (170), sakl computer system 
conprlslng a CPU (30). a memory (33), and graph- 
ics display screen (37), said method comprising the. 
steps of: 

5 

providing a pluralily of spaces (32, 50, 60 ...), 
each of said spaces comprising a graphic back- 
ground Image; 

organising said spaces (32, 50, 60 ...) in a di- 
10 rected graph structure, said spaces connected 

together with a plurality of portals (47, 48, 49, 
62,54. ...); 

- kx»ling saM plurality of portals (47. 48, 49 ...) 
In saw spaces (32. 50. 60...), each of saM por- 
ts tals having an associated target space such 
that saki portal connects the space saki portal 
Is In with saki associated target space, each of 
sakJ portals comprising a graphte k»n; 
rendering a graphic background knage of a first 
20 space and portals located In saki first space on 
said graphics display screen, said first space 
having at least a first portal; 
recehfing input from a user, saki Input compris- 
ing a portal selected from said portals k>cated 
25 in saki fust space; and 

rendering a graphk: background image of a tar- 
get space associated with saki selected portal 
on said graphics display screen (37); such that 
said user can move between spaces In said di- 
30 rected graph stmcture by selecting said portals 

within said spaces; and 
transferring selected objects In the first space 
to the target space using a second portal, saki 
interface conveying state infomiation to the us- 
35 er by animating a visual appearance of saM 

second portai to catch the users attention. 

2. The method of provWing a graphteal user interface 
on a computer system (1 70) as claimed in claim 1 , 
40 further wherein said steps of rendering a back- 
ground Image further comprises the steps of: 

rendering a first subset of saki background inv 
age on said graphics display screen (37). saki 
45 first subset from a first tocatton In saki back- 

ground image; 

receiving cfirectk>nal input from saki user, saki 
directional Input comprising a movement direc- 
tion; and 

50 rendering a second subset of saki background 

Image on said graphk» display screen (37), 
said second subset from a second location in 
said background image, said second location 
being saki first tocation plus an offset corre- 

55 spending to said movement directk>n; 

such that such user can move about within a 
bad^round Image associated with a space. 
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3. The method of providing a graphical user interface 
on a computer system (1 70) as cialmed in ciaim 2 
further wherein said step of receiving directional in- 
put from said user further comprises detecting a fin- 
ger of said user sliding across said display screen 5 
(37). 

4. The method of providing a graphical user interface 
on a computer system (170) as claimed in claim 1 
further comprising the steps oi: 

providing a wayback portal (45), said waybacic 
portal displayed in each of said spaces (32» 50, 
60...), said wayt)ack portal (45) comprising a 
stacic of data structures, each of said data 
structures comprising a pointer, each of said 
pointers pointing to one of said spaces; 
pushing a first data structure on said wayback 
portal stack when said user nK>ves to said first 
space, said first data structure containing a ^ 
pointer to said first space; 
popping a second data structure off of saM way- 
back portal stack when sakj user selects said 
wayback portal, said second data structure 
containing a pointer to a second space; and ^ 
moving said user to said second space pointed 
to by said pointer in said second data structure 
popped off said wayback portal stack; 
such that saM user can trace back atong a path 
of spaces which said user travels through. so 

5. The n^ethod of providing a graphical user Interface 
on a computer system (170) as claimed in claim 4, 
wherein said data structures In saki wayback portal 
(45) further comprises a coordinate, saki coordinate ds 
Identifying a location in said first space where saki 
user was when said user left said first space. 

6. The method of providing a graphlcai user interface 

on a computer system (170) as dalnned in claim 4, 40 
wherein said wayback portal (45) is rendered as a 
snapshot of a space pointed to by a pointer in a data 
structure on top of said wayback portal stack. 

7. The method of providing a graphical user Interface 45 
on a computer system (170) as clakned in dalm 1, 
further comprising the steps of: 

provkfing a plurality of button objects (61, 62, 
66, 67, 68, 69), each of said button objects as- so 
socidted with a function; 
rendering said button objects within saM spac- 
es, said button objects comprising graphic 
icons; and 

executing the function associated with a button ss 
object when saM button object Is selected by a 
user. 
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8. The method of provMing a graphteal user interface 
on a computer system (1 70) as claimed in daim 1 , 
wherein said display screen (37) comprises a touch 
sensitive display screen. 

9. The nnethod of providing a graphk»l user interface 
on a computer system (1 70) as dabned in daim 8 
wherein saM Input from saki user comprises aflnger 
touching sakJ touch sensitive display screen (37). 

10. The method of providing a graphical user interface 
on a computer system (1 70) as claimed In daim 1 , 
said method further comprising the steps of: 

rendering an anthropomorphb graphic charac- 
ter on said display; 

attracting said user's attention by animating 
saM anthropomorphic graphic character; and 
conveying state information to said user by 
changing a visual appearance of said anthro- 
pomorphic character. 



PatentanspiOehe 

1 . Ein Yerf ahren zum Bereltstelien einer graphbchen 
Benutzerschnittstelie auf einem Computersystem 
(170), wobei das Computersystem eine CPU (30), 
einen Speteher (33) und einen Graphikanzeigebikl- 
schlmi (37) aufweist, wobei das Verfahren die 
Schritte umfaBt: 

Berettsteilen einer Mehrzahl von RSumen (32, 
50, 60 ...), wobei Jeder der Rdunne ein graphl- 
sches Hintergruncft)ikJ auhwelst; 
Organisleren der Rdume (32, 50, 60 ...) in einer 
gertohteten Graphenstruldur. wobei die Rftume 
miteinander Qber eine Mehrzahl von Portalen 
(47. 48, 49, 52, 54, ...) verbunden sind; 
Lokalisieren der M^ahl von Portalen (47, 48, 
49 ...) in den Rftumen (32, 50, 60 ...), wobei Je- 
des der Portale einen zugehdrigen Zielraum 
derart aufweist. daB das Portal den Raum, in 
dem es sk:h befindet, mit dem zugehdrigen 
Zielraum verblndet, wobei Jedes der Portale ein 
graphisches k:on aufweist; 
Wiedergeben eines graphischen Hintergrund- 
bllds. eines ersten Raums und eines in dem er- 
sten Raum angeordneten Portals auf dem Gra- 
phikanzelgebildschimi, wobei der erste Raum 
wenlgstens ein erstes Portal auhveist; 
Empfangen einer Eingak)e von eInem Benutzer, 
wobei die Eingabe ein aus den in dem ersten 
Raum angeordneten Portalen ausge^wShites 
Portal umfaat; und 

Wiedergeben eines graphischen Hintergrund- 
bfldes eines dem ausgewfihiten Portal zuge- 
ordneten Zielraums auf dem Graphlkanzeige- 
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blidschirm (37); so daB sich der Benutzer zwt- 
schen den RSumen in der gerichteten Gra* 
phenstrulctur bewegen kann, fndem erdie Por- 
tale In den Rdumen auswShtt; und 
Obertragen ausgew&hlter Objekte in d^ er- 5 
8ten Raum in den Zletraum unter Verwendung 
eines zwetten Portals, wobel die Schnittsteile 
Zustandsinformationen an den Benutzer Qber- 
mittett, indem sie ein visuelles Erschelnungs- 
blkJ des zweiten Portals animlert, urn die Auf* 
mericsamkeit des Benutzers darauf zu lenken. 

2. Das Verfehren zum Bereitsteilen ebier graphischen 
Benutzerschnlttstelle auf e\n&n Conrputersystem 
(170) nach Anspruch 1, wobei femer die Schritte is 
des Wiedergebens eines Hinteigrundbfldes die 
Schritte umfassen: 

Wiedergeben einer ersten Teiimenge des l-lin- 
tergrundbildes auf dem Graphlkanzeigebild- ^ 
schlmi (37), wobei die erste Teiimenge aus et- 
nem ersten Ort in deno HIntergrundbikI henrOhrt; 
Bmpfangen einer Rk:htungseingabe von dem 
Benutzer, wobei die RIchtungseingabe eine 
Bewegungsrichtung umfaBt; und ^ 
Wiedergeben einer zweiten Teiimenge des Hln* 
tergrundblMes auf dem Graphikanzeigeblki- 
schimn (37), wobei die zweite Teiimenge aus ei- 
nem zweiten Ort in dem HintergrundblM her- 
rOhrt, wobei der zweite Ort dem ersten Ort plus so 
einem der Bewegungsrichtung entsprechen- 
den Versatz entspricht; 

so daB dieser Benutzer sich in einem Hinter- 
grundbiki, der einem Raum zugeordnet ist, be- 
wegen kann. ^ 

3. Das Verfahren zum Bereitsteilen einer graphischen 
Benutzerschnlttstelle auf einem Computersystem 
(1 70) nach Anspruch 2, wobei femer der Schritt des 
Empfangens einer RIchtungseingabe von dem Be- 40 
nutzer das Erfassen des Gleftens eines Fingers des 
Benutzers Qber den Anzelgebikischinm (37) um- 
faBt. 

4. Das Verfahren zum Bereitsteilen einer graphischen ^ 
Benutzerschnlttstelle auf einem Connputersystem 
(170) nach Anspruch 1. femer umfassend die 
Schritte: 

Bereitsteilen eines RCk^kwegportals (45). wobei so 
das ROckwegportal in Jedem der RAume (32, 
50. 60 ...) angezeigt wird. wobe! das ROckweg- 
portal (45) einen Stapel von Datenstrukturen 
aufweist. wobei jede der Datenstrukturen einen 
Zeiger aufweist. wobei Jeder der Zeiger auf el- ss 
nen der Rfiume verweist; 
Schieben einer ersten Datenstruktur auf den 
ROckwegportalstapel, wenn sk:h der Benutzer 



In den ersten Raum bewegt, wobei die erste 
Datenstruktur einen Zeiger auf den ersten 
Raum enth&tt; 

Wegnehmen einer zweiten Datenstruktur von 
dem ROckwegportalstapel. wenn der Benutzer 
das ROckwegportal auswdhit, wobei die zweite 
Datenstruktur einen Zeiger auf einen zweiten 
Raum enthfilt; und 

Bewegen des Benutzers in den zweiten Raum. 
auf den durch den Zeiger in der zweiten Daten- 
struktur, die von dem ROckwegportalstapel 
weggenommen worden ist, venviesen wird; 
80 daB der Benutzer entlang eines Pfedes von 
Rdum6n zurOckgefOhrt werden kann, durch die 
sk:h der Benutzer bewegt. 

5. Das Verfahren zum Bereitsteilen einer graphischen 
BenutzerBchnittBtelle auf einem ComputerBystem 
(170) nach Anspruch 4, wobei die Datenstruktursn 
In dem ROckwegportal (46) femer eine Koordlnale 
aufweisen, wobei die Koordinate einen Ort in dem 
ersten Raum Identifiziert. an dem der Benutzer war, 
als der Benutzer den ersten Raum verlieB. 

6. Das Verfahren zum BereitsteDen einer graphischen 
Benutzerschntttsteiie auf einem Computersystem 
(170) nach Anspruch 4. wobei das ROckweg;>ortai 
(45) als Momentaufnahme eines Raumes. auf den 
durch einen Zeiger in einer Datenstruktur an der 
Obersette des RQckwegportalstapels venviesen 
wird, wiedergegeben wird. 

7. Das Verfahren zum Bereitsteilen einer graphischen 
BenutzerschntitsteUe auf einem Computersystem 
(170) nach Anspruch 1, femer imifassend die 
Schrttte: 

Bereitsteilen einer Mehrzahl von Schaltfld- 
chenobjekten (61 , 62. 66, 67. 68. 69), wobei Je- 
dem der SchaltflSchenobJekte eine Funktton 
zugeordnet ist; 

Wiedergeben der Schaltnichenob|ekte In den 

Rdumen, wobei die Schaltfi&chenobjekte gra- 

phische k»ns umfassen; und 

AusfQhren der Funktion, die einem Schaltflfi- 

chenobjekt zugeordnet ist, wenn das Schaltflfi- 

chenobjekt von einem Benutzer ausgewfihit 

wird. 

8. Das Verfahren zum Bereitsteilen einer graphischen 
Benutzerschnlttsteiie auf einem Computersystem 
(170) nach Anspruch 1, wobel der Anzeigeblid- 
schirm (37) ein berOhrungsempflnditeher Anzebe* 
bildschimi ist 

9. Das Verfahren zum Bereitsteilen einer graphischen 
Benutzerschnlttstelle auf einem Computersystem 
(1 70) nach Anspruch 8, wobei die Eingabe von dem 
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Benutzer eine RngerberOhrung des beruhnings- 
empfindlichen Anzelgebildschlrms (37) umfaBt. 

10. Das Verfahren zum Bereltstellen einer graphischen 
Benutzerschnfttsteile auf einem Computersystem 5 
(170) nach Anapruch 1 , wobei das Verfehren femer 
die Schritte umfaSt: 

Wiedergeben eines anthropomorphischen gra- 
phischen Zelchens auf der Anzelge; 10 
Wecken der Aufmerlcaamkeit des Benutzers, 
Indem das anthropomorphische graphische 
Zeichen animiert wird; und 
Obermitteln von Zustandslnf oimationen an den 
Benutzer, indem das visueiie Erscheinungsbild 
des anthropomorphischen Zelchens gefindert 
wIrd. 



Revendications ^ 

1. Un proc6d6 pour foumir une interface utiiisateur 
graphique sur un syst&me infomiatique (170), ledlt 
systdme infonmatique 6tant compos6 d'une unit6 
centTBle (30), d'une memoirs ^)etd'un6crand'af- ^ 
fichage graphique (37), ledit proc6d6 comprenant 
las stapes de : 

foumituie tfune plurality d'espaces (32, 50, 
60...)i chacun desdits espaces comprenant so 
une Image graphique d'arridre-pian ; 
organisation desdits espaces (32, 50, 60...) en 
une structure de graphe orients, lesdits espa- 
ces ^t connect6s ies uns aux autres par une 3. 
pluralltd de portalls (47. 48. 49. 52, 54...) ; ^ 
localisation de ladite plurailtd de portalls (47, 
48, 49...) dans lesdits espaces (32, 50, 60...), 
chacun desdits portalls ayant un espace cft>le 
assodd de telle sorts que ledit portail connecte 
respace dans lequel se trouve tedit portail audit 40 
espace dble assodd, chacun desdits portalls 4. 
comprenant une icAne graphique ; 
rendu d'une image graphique tfanl^re-pian 
d'un premier espace et des portalls sttuds dans 
ledlt premier espace sur ledlt 6cran d'affichage ^ 
graphique, ledit premier espace ayant au nrtoins 
un premier portail; 

reception d'une entree provenant rfun utiiisa- 
teur, ladite entree comprenant un portail s6iec- 
tionn6 pamni lesdits portalls 8ltu6s dans ledit so 
premier espace ;et 

rendu d'une Image graphique d'anidre-plan 
d'un espace dble associd audit portaO s^lec- 
tionnd sur ledit teran d'affichage graphique 
(37) : de teUe sorte que ledlt utiiisateur puisse 95 
se d^lacer entre Ies espaces dans ladite struc- 
ture de graphe orients, en sdiectlonnant lesdits 
portalls au sein desdits espaces ; et 



transfert d'objets s6Iectionn^ du premier es- 
pace vers respace dble via un deuxl^me por- 
tail, ladite Interface communlquant des Infor- 
mations d'^ k rutlBsateur en animant une ap- 
parence visueiie dudit deuxidme portaB pour at- 
tlrer rattentbn des utillsateurs. 

procddd pour foumir une Interface utiiisateur 
graphique sur un systems informatlque (1 70) selon 
la revendlcation 1, dans lequel lesdites ^apes de 
rendu d'une image d'anidre-plan comprennent en 
outre Ies 6tapes de : 

rendu d'un premier sous-ensemble de ladite 
Image d'arri^re-pian sur ledlt 6cran d'affichage 
graphique (37), ledit premier sous-ensemble 
provenant d'un premier emplacenient dans la- 
dite image d'anidre-plan ; 
r6ception d'une entrSe directionnelle dudit uti- 
iisateur. ladite entrte directionnelle compre- 
nant une direction de ddplacement ; et 
rendu d'un deuxidme sous-ensemble de ladite 
image d'arri^e-plan sur ledit 6cran d'affichage 
graphique (37), ledit deuxidme sous-ensemble 
provenant d'un deuxi^me emplacenf)ent dans 
ladite Image d'arri^re-plan, ledit deuxidme em- 
placement 6tant ledlt premier emplacement 
plus un d^calage correspondant ft lacfite direc- 
tion de d^lacement ; 

de telle sorte que cet utiiisateur peut se d^la- 
cer au sein de rimage d'arrl^rei^lan associ^e 
kun espace. 

Le proc^ pour foumir une interface utiiisateur 
graphique sur un syst&me infonratique (1 70) selon 
la revendlcation 2, dans lequel, en outre ladite 6ta- 
pe de riceptlon d'une entr§e dlrectlonneOe dudit uti- 
iisateur comporte la detection d'un dolgt dudit utiii- 
sateur se d^pla^ant sur ledit dcran d'affichage (37). 

Le proc6d6 pour foumir une interface utiiisateur 
graphique sur un syst^ Informatique (1 70) selon 
la revendlcation 1 , comprenant en outre Ies dtapes 

de: 

f oumiture d'un portail de retour (45), ledit portail 
de retour est affichd dans chacun desdits es- 
paces (32, 50. 60...), ledit portail de retour (45) 
comprenant une pile de structures de donndes, 
chacune desdites structures de donndes com- 
prenant un pointeur, chacun desdits polnteurs 
pointant sur un desdits espaces ; 
enipllage d'une premiere structure de donn6es 
sur ladite pUe de portail de retour lorsque ledit 
utiiisateur se d^place vers ledit premier espa- 
ce. ladite premiere structure de donndes con- 
tenant un pointeur vers ledlt premier espace ; 
extraction d'une deuxl&me structure de don- 
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ntes de ladite pile do portail de retour lorsque 
ledit utilisateur 86lectionne ledtt portail de re- 
tour, ladlte deuxi^me structure de donn^es 
contenant un pointeur vers un deuxidme 
espace ; et ^ 
d^lacement dudit utilisateur vers ledit deuxi6- 
me espace sur lequel pointe ledit pointeur dans 
iadite deuxi^me structure de donnto extraite 
de Iadite pile de portail de retour ; 
de telle sorte que ledit utilisateur peut retracer io 
un itindraire des espaces traversds par ledit uti- 
lisateur. 

5. Le proc^ pour foumir une interf^ utilisateur 
graphique sur un syst^e Infonnatique (1 70) selon 
la revendication 4, dans lequel lesdites structures 
de donn^ dans ledit portail de retour (45) com- 
prend en outre des coorck>nn6e8, lesdites coordon- 
n6es identtflant un emplacenient dans ledit premier 
espace oCk ledit utilisateur se trouvait lorsque ledit ^ 
utilisaleur a quitt6 ledit premier espace. 

6. La proc6d6 pour foumir une interface utilisateur 
graphique sur un systdme infomnatique (170) selon 

la revendication 4. dans lequel ledit portail de retour ^ 
(45) est rendu sous la forme tfun Instantand d*un 
espace sur lequel pointe un pointeur dans une 
stnicture (to donn^ au sommet de Iadite pile de 
portail de retour. 

30 

7. Le proc6d6 pour foumir une Interface utilisateur 
graphique sur un systdme inf onnatique (1 70) selon 
la revendication 1 , comprenant en outre les 6tapes 
de: 

35 

foumiturs d*une plurality d'objets k>outon (61, 
62, 66, 67, 68. 69), chacun desdits objets bou- 
ton dtant associd h une fonctlon ; 
rendu desdits objets bouton dans lesdits espa- 
ces, lesdits objets bouton comprenant des led- ^ 
nes graphlques ; et 

extortion de la fonction associ6e k un objet 
bouton brsque ledit obJet bouton est s6lectf on- 
n6 par un utilisateur. 

45 

8. Le proc^ pour foumir une interface utilisateur 
graphique sur un syst^me informatlque (170) selon 
la revendication 1 , dans lequel ledit 6cran d'afficha- 
ge graphique (37) comporte un toan tactile. 

50 

- 9. Le proc6d6 pour foumir une interface utilisateur 
graphique sur un systtoe informatlque (1 70) selon 
la revendication 8. dans lequel Iadite entree dudit 
utilisateur comporte un dolgt touchant ledit 6cran 
tactile (37). " 

10. Le proc6d6 pour foumir une interface utilisateur 
graphique sur un systems Informatique (1 70) selon 



la revendication 1, ledit proc6d6 comprenant en 
outre les 6tapes de : 

rendu d'un personnage graphique anthropo- 
morphique sur ledit 6cran ; 
animation dudit personnage anthropomorphl- 
que pour attirer rattention dudit utilisateur : et 
communication ^informations d'etat aucfit utili- 
sateur en modiflant une apparence visuelle du- 
dit personnage anthropomorphlque. 



25 



EP0626635B1 




26 



EP06266ffiB1 



170 



MBMORT 



•83 







CPU 



'30 



/ 1 



INPUT/OUTPUT 



•34 



OOMUnmCATlOH 



\m ^ mm ^ m. 



u 
VCB 



3 




Other INPUT 

DEVICES 

(MOUSEk 
TRACKBALL^ 
JOYSTICK) 



J 



-37 



TOUCH 
SCRBEK 
DISPLAY 



THERMOSTAT 




160 



Figure lb 



27 



EP0626635B1 



401 



403 



404 



Commuaication 



CPU 



Bfemoiy 



Device Control 



405 



Figure Ic 



-160 



420 



424 



Emitter 



Recover 



■ Memoiy 



Device Control 
Circmtty 



421 425 

Figure Id 



-155 



460 



465 



Receiver 



Device Control 
Circuitiy 



-167 



Figure le 

(PriorArt) 



28 



EP0626635B1 



EARTH 





TOUBTV 



iiil 



AMOTBERTOWN 



ffni iiinsiii BZ8 
BZi [||pt|L Ba 










i 




YOUR VCR 



Figure2a 



29 



EP0626635B1 




30 



EP0626G35B1 




31 



EP 0626635 B1 




32 



EP0626G35B1 




EP0626G35B1 




34 



EP0626635B1 




Murder,S he Spoke | 

WKile Jessico f Ang«ld Lansbary) is recording one 
of her novels for the Mystery Books for the Blind 
serlesy a murder occurs With Michael Cotei 
Charlie Daniels and Jonna Lee • 

Oiyihour* 

45- 




41 



) / 




35 



EP0626635B1 




Figure 11 



36 



EP0626G35 



B1 




37 



EP0626635B1 




36 



EP0K6635B1 




EP0626 



635 B1 




40 



EP0626635B1 




41 



EP0626 635B1 




Figure 17a 



Figure 17h 



Figure 17c 



42 



EP0626G35B1 




Figure 18a 



Figure 18b 




ISO 



THBPMggTM- 



r 



i 



(memoby) 

40S_ZZ]_1 
"^ i CPU i 



40 



I 



OOMM 



Figure 18c 



ISO 



JBSBMQ3EAZL 



404 

I 



- (MEMORYj 



Figure 18d 



■170 




^170 




43 



EP0626635B1 



150— 
SOI 



-A 



— 170 



\ 

160 



807 

Figure 19a 



160— 
301< 



-A 



^808 



3U- 



160 



f 

309 



* \ 



307 

Figure 19b 




313- 



N 

160 



A 



307 309 

Figure 19c 



ISO— 
301 



— 170 
-311 



— 170 
-311 





^303 

313— 

s ^ 






^ 160 





—170 
-311 



307 309 

Figure 19d 



44 



EP0626635B1 



REMOTE DEVICE IS TUBNED ON 
AND BEGINS ANNOUNCINO 
IT EAS AN INTERFACB TO EXPORT 



— 101 



DISPLAY DEVICE RECEIVES EXPORT OF 
INTERFACE NOTICE FROM THE DEVICE 
AND SIGNALS BACK TO BEGIN TRAN8MITTIN0 



— 102 



I 



REMOTE DEVICE BROADCASTS PROGRAM 
OBJECTS FOR ITS INTERFACE 



— loa 



I 



DISPLAY DEVICE RECEIVES PROGRAM OBJECTS 
FROM REMOTE DEVICE AND DISPLAYS OBJECT 
ON THBDI8PLAY SCREEN 



— 104 



I 



DISPLAY DEVICE INVOKES REMOTE D EVICE S 
USER INTERFACE PROGRAM OBJECT WHEN 
REMOTE DEVICE DISPLAY OBJECT SELECTED 



— 106 



I 



USER INTERACTS WITH REMOTE 
DEVICES GUI ON DISPLAY DEVICE 



— 106 



I 



CONTROL METHOD INVOKED ON CORRECT 
REMOTE DEVICE AS INDICATED BY REMOTE 
NAME SERVER, TO CONTROL REMOTE DEVICE 



107 



I 



REMOIE DEVICE CHANGES IN RESPONSE TO 
CONTROL METHOD INVOCATION 



— 108 



RETURN VALUES, IF ANY, ARB BROADCAST BACK 
TO THE DISPLATDEVICB 



— 109 



Figure 20 



45 



EP0626635B1 



FINITE STATE MACHINE 
FOR THE REQUESTINQ SIDE OF A 
BBMOTB METHOD INVOCATION 




Figure 21a 



46 



EP0626e35B1 



FINITB STATE MACHINB 
FOR THE RECEIVING SIDE 
OF A REMOTE METHOD 
INVOCATION 



TRANSMIT 
rM WORKING 
461 



JTBS 



BBINQ 
WORKED ON? 
457 



W ATT FO R 
NEXT 



NO. ALREADY 
COMPLETED 
AND REPLIED^ 
TO 



RETRANSMIT 
REPLY 
469 



SOBIET trofQ I 
HAVBI^EN 
BEFORE 



WAITING TO 
RECEIVE 
451 



REC EIVE 
PACKET 



WHAT IS 
IT? 
453 



TRANSMIT 



SOMETHING^ 
NEW 



Figure 21b 



EXECUTE 
REQUEST 
455 



47 



EP0626635B1 




EP0626635B1 



155- 



424 



- CMEMORY) 



I 



421) 



420 >^ 



162 




Figure 22a 



Figure 22b 



THERMOOTAT 



- CMEMORY) 



156- 



1 



431 



420\ 




424 



Figure 22c 



THERMOSTAT 



424 



— (MEMORY) 



166- 



i 



420\ 



421 



RECBIVERj 



170 




Figure 22d 



49 



EP0626635B1 



363 




Figure 23a 



351- 



420 



431 

Figure 23b 



185— 



353 

Si. 



420 



361- -^V 



3 



421 

Figure 23c 



156— 



353 

SZ 



351- -^V 



C — 430 

4 



431 

Figure 23d 



50 



EP0626635B1 



REMOTE DEVICE'S USSR INTERFACE AND CONTROL 
PROGRAM CODE TRANSFERRED TO 
DISPLAY DEVICB 



—134 



I 



DISPLAY DEVICE INVOKES USER 
DiTBRFACE PROGRAM CODE 



I 



—125 



DISPLAY DEVICB DISPLAYS REMOTE DEVICE'S 
GRAPHIGAL USER INTERFACE 



I 



—126 



USER INTERACTS WITH DBVICE1S GUI ON DISPLAY 
DEVICE, WHICH CAUSES A CONTROL METHOD 
TO BE INVOKED IN THE DISPLAY DEVICE 



I 



—127 



METHOD INVOCATION RESOLVES THE USERS 
ACTIONS INTO A CONTROL CODE FOR 
TBB REMOTE DEVICE 



I 



—128 



CONTROL CODE IS TRANSMITTED TO REMOTE 
DEVICE VIA SOME COMMUNICATION BIEANS 



—129 



1 



REMOTE DEVICE CHANGES IN RESPONSE TO 
THE CONTROL CODE EXECUTION 



—130 



I 



RETURN VALUES, IF ANY, ARE BROADCAST BACK 
TO THE DISPLAY DEVICE 



—131 



Figure 24 



51 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: ^ 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



